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FOREWORD 

a.  This  technical  report,  BDM/A-85-Q510-TR,  is  submitted  by  The 
BOM  Corporation,  1801  Randolph  Road,  S.E.,  Albuquerque,  New  Mexico 
87106,  to  the  Air  Force  Operational  Test  and  Evaluation  Center, 
Kirtland  Air  Force  Base,  New  Mexico,  87117.  This  report  is  in  com¬ 
pliance  with  CDRL  Item  A008,  Contract  F29601-80-C-0035,  and  fulfills 
the  requirements  of  paragraph  7.3  of  Subtask  Statement  327/04,  titled 
"Risk  Profile  Development  for  Software  Supportabi 1 ity. " 

b.  This  report  is  the  result  of  effort  by  Mr.  Walter  Huebner,  Jr. 
(Task  leader).  Dr.  David  Peercy  (Technical  Leader),  Mr.  M.  Donan 
Est ill,  Jr.  and  Ms.  Jean  C.  Wu  of  The  BDM  Corporation.  The  primary 
Subtask  Statement  Project  Officer  is  Capt.  Eric  H.  Tomlin 
( AF0TEC/LG5T) ;  the  alternate  Subtask  Statement  Project  Officers  are 
Maj.  Gary  R.  Horlbeck  (AF0TEC/LG5T)  and  Mr.  Jim  M.  Baca  (AF0TEC/LG5) . 


Reviewed  and  approved  by: 


Walter  F.  Huebner 
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SECTION  I 
INTRODUCTION 

1.1  BACKGROUND. 

a.  The  Air  Force  Operational  Test  and  Evaluation  Center  (AFOTEC) 
has  the  responsibility  for  conducting  operational  test  and  evaluation 
(OT&E)  of  assets  entering  the  Air  Force  inventory.  AFOTEC  has 
developed  and  implemented  various  software  OT&E  methodologies.  These 
methods  have  matured  and  have  become  the  Air  Force  standard  for  eval¬ 
uating  software  supportabil ity.  Each  of  these  developed  methods 
evaluates  specific  characteristics  of  the  supportabil ity  aspects  of 
delivered  software  and  software  support  resources.  These  stand-alone 
evaluations  provide  AFOTEC  with  information  to  identify  particular 
software  supportabil ity  deficiencies,  but  do  not  identify  overall 
risk  associated  with  contractor  or  military  ownership  and  organic 
maintenance  of  contractor-del i vered  software. 

b.  Assessing  the  software  supportabi 1 ity  risk  of  Air  Force 
acquired  systems  is  necessary  to  enable  various  decision  makers  to 
properly  plan  for  system  deployment.  Risk  assessment  (RA)  is 
required  throughout  the  system  acquisition  life  cycle.  The  perspec¬ 
tive  of  OT&E  is  focused  upon  the  overall  system  mission  operation, 
including  support.  Methods  are  needed  to  point  software  testers  to 
areas  that  require  testing  emphasis,  and  provide  decision  makers  with 
an  assessment  of  the  software  supportabil ity  risk. 

c.  Since  major  weapon  systems  are  using  more  sophisticated 
computer  applications,  software  support  for  these  systems  is  becoming 
an  increasingly  greater  cost  factor  in  overall  system  cost.  Further¬ 
more,  when  many  enhancements  to  a  system  are  dependent  on  software 
modifications,  the  timeliness  of  such  software  support  is  critical  to 
system  operational  availability  and  effectiveness.  Because  of  this 
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criticality  of  the  software  support  function  to  overall  system 
mission  operational  capability,  it  is  desired  that  top  decision 
makers  be  aware  of  the  risk  associated  with  the  software  support- 
ability  of  a  system  at  specific  points  during  the  system  development 
cycle,  but  in  particular  at  the  conclusion  of  OT&E. 

d.  In  order  to  determine  this  risk  during  OT&E,  AFOTEC  initiated 
the  development  of  a  risk  assessment  methodology  of  software 
supportabi 1 ity  with  emphasis  on  system  mission.  This  approach  was 
taken  to  provide  more  meaningful  information  to  top  level  decision 
makers  than  had  previously  been  available.  In  1983,  AFOTEC  produced  a 
concept  proposal  (reference  8.7)  for  computer  resources  risk  assess¬ 
ment  during  operational  test  and  evaluation.  This  effort  integrated 
an  approach,  appropriate  models,  and  subjective  and  quantitative 
software  operational  and  supportabi 1 ity  measures  into  a  management  - 
oriented  assessment  of  user  and  supporter  risk.  This  initial 
involvement  with  the  application  of  risk  assessment  to  software  sup- 
portability  provided  AFOTEC  with  justification  to  support  a  study  of 
the  feasibility  of  development  and  implementing  a  risk  assessment 
methodology  for  software  supportabi 1 ity  (RAMSS).  The  feasibility 
study  (reference  8.8)  proposed  a  conceptual  RAMSS  which  incorporated 
a  theoretical  foundation  for  risk  assessment  with  the  software  evalu¬ 
ation  tools  presently  being  used  by  AFOTEC.  The  risk  assessment 
methodology  represents  the  authors'  determination  of  the  most  prac¬ 
tical  way  to  assess  risk  under  the  criteria  and  constraints  with 
which  AFOTEC  must  work  and  the  software  evaluation  process  in 
general.  The  results  of  the  feasibility  study  are  reported  in 
references  8.1,  8.2,  and  8.3. 

e.  Given  that  the  feasibility  study  demonstrated  the  merit  of 
implementing  the  proposed  RAMSS,  the  next  logical  step  in  the  devel¬ 
opment  of  the  model  has  been  to  gather  and  analyze  data  on  software 
support  activities  for  systems  of  interest  to  the  Air  Force,  and 
develop  a  basis  from  which  measurement  of  risk  can  be  made.  This 
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report  documents  the  results  of  the  data  collection  and  analysis 
step. 


1.2  STUDY  OBJECTIVE. 

a.  The  overall  objective  of  this  task  study,  as  stated  in  Subtask 
Statement  327/00  (reference  8.0)  was  "to  develop  historical  profiles 
of  software  support  activities  from  USAF  software  support  facilities. 
These  profiles  shall  provide  AFOTEC  with  baselines  against  which 
evaluations  of  software  support  risk  can  be  developed."  Various 
types  of  data  and  information  have  been  collected  from  approximately 
80  major  weapons  systems  and  subsystems  during  this  study.  These 
data  have  formed  the  basis  for  the  development  of  historical 
profiles,  or  "baselines,"  from  which  risk  can  be  measured.  These 
profiles  are  presented  in  section  V  of  this  report. 

b.  While  the  above  objective  is  satisfied  in  this  report,  there 
are  other  benefits  which  are  fallouts  of  the  process  of  performing 
the  study  and  collecting  the  data.  These  other  benefits  include: 

(1)  An  understanding  of  the  kinds  of  software  maintenance 
activity  data  available  at  Air  Force  software  support 
facilities 

(2)  Development  of  a  survey  format  that  will  be  used  to  set  a 
standard  for  collecting  software  maintenance  activity 
data  from  other  systems  or  from  the  same  systems  in 
future  data  collection  efforts 

(3)  A  study  of  how  the  types  and  availability  of  exoected 
software  maintenance  activity  data  affect  the  applic¬ 
ability  of  the  recommended  RAMSS 
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(4)  As  a  result  of  collecting  the  data  from  different  loca¬ 
tions,  a  common  information  data  base  has  been  formed 
from  which  implications  of  systemic  improvements  to 
software  maintenance  support  activities  may  be  inferred, 
with  an  eye  toward  improving  overall  capabilities  Air 
Force  or  OoO-wide. 

1.3  REPORT  ORGANIZATION. 

a.  The  remainder  of  this  report  is  organized  into  seven  addi¬ 
tional  sections,  plus  a  set  of  appendices  that  include  detailed 
information  supporting  the  results  of  this  study.  Report  sections 
satisfy  the  following  objectives: 

(1)  Section  II  contains  an  executive  summary  of  this  study, 
and  a  technical  summary  which  includes  an  overview  of  the 
approach,  a  summary  of  the  analysis  conducted,  an 
historical  profile  example,  a  review  of  the  methodology, 
and  conclusions/recommendations  from  this  study. 

(2)  Section  III  contains  a  general  discussion  of  the  approach 
taken  in  performing  this  study.  Specific  topics  include 
assumptions  concerning  the  software  maintenance  activity, 
the  site  survey  format,  facility  visit  procedures,  a  list 
of  the  facilities  visited  and  systems  examined,  and  a 
recommended  maintenance  data  collection  form  for  software 
support  sites. 

(3)  Section  IV  describes  results  from  data  analyses. 

(4)  Section  V  contains  the  historical  profiles:  one  for  all 
systems,  and  other  profiles  by  site,  and  by  system  type. 
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(5)  Section  VI  briefly  describes  the  RAMSS  and  gives  an 
example  of  how  to  use  the  historical  profiles  to  predict 
software  supportabi 1 ity  risk. 

(6)  Section  VII  contains  a  discussion  of  the  conclusions  and 
recommendations  from  this  study. 

(7)  Section  VIII  lists  the  documents  whose  contents  have 
formed  the  basis  for  this  report. 

(8)  Appendix  A  lists  acronyms  used  in  this  report. 

(9)  Appendix  B  is  a  glossary  of  terms  used  in  this  report. 

(10)  Appendix  C  is  the  final  version  of  the  site  survey  format 
recommended  for  future  use. 

(11) -  Appendix  D  contains  the  "raw"  data  collected  during  this 

study.  This  data  forms  the  basis  from  which  the  histori¬ 
cal  profiles  were  derived. 

(12)  Appendix  E  contains  systems  descriptions  for  the  software 
systems  studied. 

(13)  Appendix  F  contains  all  trip  reports  filed  as  a  result  of 
this  study. 
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SECTION  II 

SUMMARY 

This  section  is  separated  into  two  parts.  The  first  part  is 
an  executive  summary  describing  the  evolution  of  software  support- 
ability  risk  assessment  within  the  Air  Force  Operational  Test  and 
Evaluation  Center ‘ (AFOTEC) .  The  second  part  is  a  technical  summary 
of  the  results  from  the  current  task  to  provide  AFOTEC  personnel  with 
data  to  support  the  methodology  to  assess  software  supportabi lity 
risk. 

2.1  EXECUTIVE  SUMMARY 

2.1.1  Concept  Development. 

a.  Since  1982,  AFOTEC  has  been  analyzing  the  problem  of  how  to 
assess  the  risk  to  the  Air  Force  of  supporting  software  acquired  for 
weapon  systems.  A  concept  for  computer  resources  risk  assessment 
during  operational  test  and  evaluation  was  proposed  in  1983 
(reference  8.20).  Several  issues  evolved  from  this  proposal.  First, 
the  assessed  risk  should  reflect  software  supportabi lity  impact  upon 
the  system  at  a  level  appropriate  for  AFOTEC  reporting  requirements. 
Second,  supportabi 1 ity  is  a  concern  for  both  the  user  and  the 
supporter.  Any  defined  risk  of  software  supportabi 1 ity  should 
reflect  some  aspect  of  user  risk  and  supporter  risk.  Third,  current 
AFOTEC  methods  of  evaluating  software  supportabi 1 ity  should  be  inte¬ 
grated  into  the  risk  assessment  method.  Also,  the  risk  assessment 
method  should  be  adaptable  to  include  other  AFOTEC  concerns  such  as 
software  maturity  and  software  reliability. 

b.  This  initial  concept  proposal  provided  AFOTEC  with  justifica¬ 
tion  to  study  the  feasibility  of  developing  and  implementing  a  risk 
assessment  methodology  for  software  supportabi 1 ity  (RAMSS).  The 
approach  for  this  study  (references  8.1,  8.2,  3.3)  included: 
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(1)  Literature  review  and  assemblage  of  a  data  base  of  rele¬ 
vant  tools,  techniques  and  methods 

(2)  Analysis  of  relevant  tools,  techniques,  and  methods  for 
feasibility  of  application  to  AFOTEC's  needs 

(3)  Development  of  a  framework  for  assessing  software  sup- 
portability  risk  along  with  a  preliminary  set  of  risk 
measures. 

c.  The  primary  conclusion  from  this  feasibility  study  was  that  a 
RAMSS  could  be  developed  based  upon  the  framework  derived  as  part  of 
the  study.  However,  there  were  still  several  technical  issues  which 
needed  to  be  resolved.  Of  these  issues,  the  major  one  concerned  the 
need  to  establish  a  baseline  against  which  to  measure  risk.  Since 
risk  is  defined  (for  this  study)  as  "the  potential  for  realization  of 
unwanted,  negative  consequences  of  an  event,"  it  is  necessary  to  have 
a  baseline  of  software  support  activities  in  order  to  tell  when  a 
consequence  may  be  negative.  This  baseline,  called  an  historical 
maintenance  profile,  reflects  how  software  support  resources  are 
being  used  to  perform  the  software  support  activities.  Given  this 
information,  the  framework  recommended  by  the  feasibility  study  could 
be  used  to  compute  measures  of  risk  and  incorporate  the  issues 
proposed  in  1983. 

2.1.2  Methodology  Requirements  (Inputs).  Figure  2-1  illustrates 
interfaces  with  the  proposed  risk  assessment  methodology  for  software 
supportability  (RAMSS).  The  inputs  consist  of: 

(1)  The  historical  profile  of  software  maintenance  activity 

(2)  A  user/supporter  agreement  on  planned  software 
maintenance  changes  and  support  resource  requirements  for 
the  software  system  being  evaluated 
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(3)  An  evaluation  of  software  support  capabilities  using 

current  AFOTEC  methods. 

2.1.3  Methodology  Analysis.  The  RAMSS  inputs  are  combined  and 

analyzed  to  compute  measures  of  risk  for  the  system  being  evaluated. 

2.1.4  Methodology  Benefits  (Results). 

a.  The  major  results  of  the  RAMSS  are  also  illustrated  in 

figure  2-1.  These  results  include: 

(1)  The  risk  measure  which  quantifies  the  probability  of  the 
user/support  agreement  not  being  accomplished  with 
current  software  support  capabilities 

(2)  The  capability  to  identify  the  impact  of  the  total  system 
risk  as  high,  medium,  or  low 

(3)  The  identification  of  the  drivers  of  the  software 

supportabi 1 ity  risk 

(4)  The  projection  of  alternative  choices  for  risk  reduction 
(for  instance,  by  improving  certain  aspects  of  current  or 
projected  software  support  capabilities). 

b.  With  this  information,  the  decision  maker  can  assess  the 
effect  of  software  supportabi 1 ity  upon  system  suitability  and 
effectiveness.  In  addition,  detailed  data  are  available  to  help 
answer  specific  questions  such  as  why  particular  areas  of  software 
supportabi  1  ity  are  drivers  and  how  the  measured  risk  can  be  reduced 
to  an  acceptable  level. 

2.1.5  Concl usion.  The  main  recommendation  from  the  current  study  is 
that  the  proposed  risk  assessment  methodology  for  sof tware/support- 
ability  be  applied  in  a  closely  controlled  manner  to  an  actual 
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software  system  operational  test  and  evaluation  effort.  This  is 
necessary  in  order  to  validate  the  methodology  and  establish 
procedures  that  will  allow  AFOTEC  personnel  to  effectively  apply  the 
methodology  in  the  operational  test  and  evaluation  of  future  systems. 

2.2  TECHNICAL  SUMMARY. 

This  part  of  the  report  summarizes  the  approach  taken  in 
conducting  the  study,  describes  the  results  of  data  analysis, 
presents  a  sample  historical  maintenance  profile  developed  from  the 
data,  discusses  the  risk  assessment  methodology,  and  condenses  the 
concl us i ons/recommendat i ons . 

2.2.1  Approach  Overview. 

a.  The  basic  approach  to  gathering  the  software  support  activity 
data  necessary  to  create  historical  maintenance  profiles  consisted  of 
visiting  selected  locations  that  were  performing  the  activities  of 
interest.  Early  in  the  study  it  was  recognized  that  a  survey  form 
should  be  developed  to  permit  a  standard  method  of  collecting  these 
data.  AFOTEC  and  BOM  personnel  worked  together  to  create  the  survey 
form  and  select  the  appropriate  software  support  facilities  to  visit. 

b.  The  format  of  the  survey  form  created  to  collect  the  software 
support  data  consisted  of  preface  statements  followed  by  requests  for 
three  types  of  information:  software  background  data,  software 
assessment  data,  and  desirable  maintenance  data.  The  preface 
statements  introduced  the  purpose  of  the  study,  followed  by  defini¬ 
tions  applicable  to  the  survey.  Software  background  data  primarily 
identified  and  described  characteristics  of  the  software  system  being 
surveyed.  The  software  assessment  data  were  requested  to  determine 
the  correlation  between  the  software  evaluation  tools  (which  are 
capable  of  "measuring"  the  risk  of  supporting  software)  and  a 
subjective  assessment  of  risk  by  personnel  who  are  currently 
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supporting  the  software  of  interest.  The  final  data  requested  were 
the  actual  information  that  were  used  to  build  the  risk  profiles. 
Key  elements  included  block  release  start  and  end  dates,  estimated 
and  actual  person  effort,  number  and  types  of  software  changes  imple¬ 
mented  in  the  block  release,  and  major  problems. 

c.  Unlike  the  survey  form,  facility  visit  procedures  were  not 
formalized,  but  this  report  documents  for  future  reference  the 
methods  employed.  The  methods  are  divided  into  pre-visit  procedures, 
on-site  visit  procedures,  and  post-visit  procedures.  Pre-visit  pro¬ 
cedures  primarily  included  initial  telephone  contact  with  the 
personnel  targeted  for  the  visit,  as  well  as  mailing  the  survey  form 
in  advance  of  the  visit,  along  with  a  letter  of  confirmation  from 
AFOTEC.  Key  pre-visit  activities  also  included  the  identification  of 
appropriate  AFOTEC  personnel  to  accompany  the  survey  team,  requests 
for  early  completion  of  the  survey  form,  and  the  identification  of 
senior  software  engineers  for  the  software  systems  of-  interest. 
On-site  visit  procedures  generally^  consisted  of  a  management 
overview,  followed  by  a  briefing  given  to  senior  software  engineers 
by  AFOTEC  personnel,  and  then  a  detailed  presentation  by  BDM 
personnel  on  the  software  maintenance  activity  data  being  collected 
by  this  study.  After  obtaining  the  data  and  answering  questions,  a 
key  item  was  to  quality-check  the  data  for  completeness  and  consis¬ 
tency  before  ending  the  visit.  Post-visit  procedures  consisted  of 
organizing  the  information  received  and  following  through  on  open 
action  items  from  the  visit. 

d.  A  total  of  seven-  instc1 l ations  were  visited  during  this  study. 
A  list  of  these  installations  is  shown  in  table  2-1.  Trip  reports 
from  these  visits  are  contained  in  appendix  F  of  this  report. 

2.2.2  Results  of  Analysis.  The  goals  of  the  data  analysis  effort 
have  been  to  verify  the  data,  reduce  the  data  to  a  standard  form, 
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Table  2-1. 
Trip  Schedule 


SITE 

NORAO 

Warner  Robins  ALC 
Sacramento  ALC 
Castle  AFB 
Ogden  ALC 

♦Oklahoma  City  ALC/Tinker  AFB 
Langley  AFB 


OATE  VISITED 
7-9  January  1985 
28  January  -I  February  1985 
24-26  February  1985 
27-28  February  1985 
22-25  April  1985 
13-16  May  1985 
21-25  July  1985 


♦The  E-3A  facilities  at  Tinker  AFB  were  not  associated  with  the 
Oklahoma  City  ALC 


build  a  data  base,  assess  data  validity,  report  summary  data  infor¬ 
mation,  build  the  historical  maintenance  profiles,  and  perform 
statistical  data  analysis.  These  activities  were  performed  on  the 
three  types  of  data  collected:  background  data,  evaluation 
(assessment)  data,  and  -desirable  maintenance  activity  data.  Results 
of  analysis  are  summarized  below. 
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2.2.2. 1  Background  Oata  Analysis.  Background  data  was  collected 
more  for  descriptive  completeness  than  for  thorough  statistical 
analysis.  Some  interesting  results  from  this  data  include: 

(1)  software  systems  studied  varied  from  1000  source  lines  of  code  to 
more  than  2.8  million;  (2)  the  number  of  personnel  directly 
supporting  the  software  varied  from  1  to  84;  (3)  there  is,  on  the 
average,  one  support  person  for  every  24,000  lines  of  source  code 
(with  a  data  range  from  300  lines  to  273,000  lines  per  person). 
Analysis  does  not  indicate  direct  correlation  of  background  data 
collected  with  actual  maintenance  data  collected.  For  example,  there 
does  not  appear  to  be  a  direct  correlation  of  software  system  size 
with  the  number  of  personnel  supporting  the  system. 
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2.Z.2.2  Evaluation  Data  Analysis. 

a.  Evaluation  data  was  analyzed  to  determine  relationships 

between  the  software  evaluation  tools  used  to  "measure"  the  risk  of 
software  supportabi 1 ity  and  a  subjective  assessment  of  the  risk  by 

personnel  now  supporting  the  software. 

b.  Evaluation  ratings  were  collected  on  45  categories,  of  wnich 

44  were  measures  of  the  system  supportabi 1 ity  characteristics  which 
affect  the  risk  of  not  being  able  to  support  a  given  system.  The 

45th  evaluation  rating  was  a  direct  rating  of  the  overall  risk  as 
perceived  by  the  evaluator.  Factor  analysis  of  the  software 

supportabi 1 ity  evaluation  ratings  showed  that  37  of  the  rating 
categories  (7  had  insufficient  data)  could  be  combined  into  six 
factors  interpreted  as  evaluating  basic  areas:  support 

(configuration  and  maintenance)  management,  the  product  (software  and 
documentation) t  personnel,  organization,  facilities,  and  support 
systems.  These  factors  accounted  for  79  percent  of  the  tota' 
variance  in  the  37  rating  categories;  hence  most  of  the  information 
contained  in.  the  rating  categories  could  be  condensed  into  only 
6  factors. 

c.  The  six  factors  of  software  supportabi 1 i ty  were  modeled  to 

determine  which  factors  have  significant  impact  on  the  risk.  Of  the 
six  factors,  four  were  determined  to  be  significant:  suDport 

management,  product,  personnel,  and  support  systems.  Using  these 
factors  in  the  derived  regression  mode5  was  the  best  method  of  trie 

three  methods  analyzed  for  computing  risk  from  software 
supportabi 1 i ty  evaluation  metrics. 

d.  The  other  two  models  for  computing  risk  from  software 
supportabi 1 ity  metrics  were  another  regression  model  and  a  simoie 
linear  function.  For  the  regression  model,  the  risk  estimated  by  the 
survey  site  support  personnel  was  regressed  against  the  top  level 
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software  supportability  metric.  This  model  showed  a  reasonable  fit 
to  the  plot  of  data  points.  This  model  did  not  account  for  as  much 
variance  as  the  factor  regression  model.  As  expected,  the  simple 
linear  conversion  of  software  supportability  evaluation  metric  to 
risk  using  the  formula: 

Risk  =  i  Evaluation^Metric  -  1 

provided  the  poorest  fit  to  the  data. 

2. 2. 2. 3  Maintenance  Data  Analysis. 

a.  Initial  analysis  of  the  software  maintenance  data  involved 
determining  what  could  be  done  with  the  data.  This  occurred  as  a 
result  of  insufficient  knowledge  at  the  beginning  of  this  survey 
about  the  availability  of  certain  desired  critical  data  items.  No 
system  .had  all  the  information  desired  in  an  easily  accessible  form. 
Actual  person  effort  per  change  or  per  block  release  was  almost 
always  unavailable.  Nevertheless,  intensive  query  of  knowledgeable 
support  personnel  provided  reasonably  good  block  release  data.  The 
physical  capability  to  record  and  retrieve  the  desired  data  exists, 
but  there  was  simply  no  universal  standard  for  what  to  collect  or  how 
to  report  it. 

b.  Analysis  of  the  software  release  data  focused  upon  creating 
the  historical  maintenance  profiles  with  available  person-months  per 
change  as  a  key  parameter.  Because  this  parameter  was  computed  as  a 
simple  ratio  for.  each  release,  and  because  the  person-months  are 
typically  not  known  prior  to  completion  of  a  release,  it  was 
desirable  to  analyze  whether  person-months  per  change  was  strongly 
correlated  to  some  other  known  parameters.  Linear  regression 
analysis  was  conducted  with  independent  variables  being:  number  of 
source  lines,  percentage  of  correction  requests,  percentage  of  low 
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complexity  changes,  percentage  of  normal  priority  requests,  percen¬ 
tage  of  high-level  language,  and  average  skill  level  of  personnel. 
The  results  of  the  analysis  showed  no  strong  relationships.  It  is 
likely  that  the  inconsistency  and  inaccuracy  of  some  data  affected 
these  results. 

2.2.3  Maintenance  Profiles. 

a.  The  actual  historical  maintenance  profiles  generated  by  the 
maintenance  data  collected  in  this  study  are  contained  in  section  V. 
The  general  form  of  the  profiles,  which  plot  the  available  person- 
months  per  change  on  the  x-axis  and  the  number  of  releases  on  the 
y-axis,  is  shown  in  figure  2-2.  The  profile  represents  the 
number  (Y)  of  software  releases  accomplished  with  X  "unit  cost" 
(available  person-months  per  software  change  required  for  that 
release).  A  software  change  is  defined  as  a  measureable  change 
documented  by  a  software  change  request  or  a  similar  item.  Profiles 
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Figure  2-2.  Historical  Maintenance  Profile  Example 
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were  also  generated  by  software  support  site  and  software  system  type 
(e.g.,  operational  flight  program,  communications-electronics) .  The 
profiles  contained  in  this  report  thus  form  historical  baselines 
against  which  the  supportabi 1 ity  risk  of  future  systems  can  be 
assessed. 

2.2.4  Methodology  Review. 

a.  There  are  four  basic  phases  involved  in  executing  the  risk 

assessment  methodology  for  software  supportabi 1 ity:  planning, 

evaluation,  analysis,  and  reporting.  The  primary  function  in  the 
planning  phase  is  to  establish  an  appropriate  baseline  profile  of 
expected  maintenance  actions.  A  second  key  function  in  the  planning 
phase  is  to  obtain  user/supporter  agreement  on  the  "acceptable" 
support  risk  for  the  software  system  of  interest.  During  the 
evaluation  phase,  the  .software  system  supportabi 1 ity  is  "measured" 
against  this  agreement  using  the  software  evaluation  tools.  The 
third  phase,  analysis,  involves  comparing  the  "acceptable"  risk  with 
the  "measured"  risk  to  examine  differences  and  perform  trade-off 
studies  when  the  "measured"  risk  is  greater  than  the  "acceptable" 
risk.  The  final  phase  is  the  report  to  the  decision  maker  on  the 
results  of  the  analysis  which  highlight  risk  measures,  drivers, 
alternatives  to  reduce  risk,  and  overall  risk  impact. 

b.  The  primary  refinement  to  the  risk  assessment  methodology,  as 
described  in  reference  8.3,  is  in  the  level  of  detail.  The  baseline 
software  supportabi 1 ity  profile  of  change  requests  still  consists  of 
27  categories  for  the  three  software  maintenance  types,  three  levels 
of  complexity,  and  three  levels  of  priority  in  all  combinations  with 
each  category  having  two  values:  time  to  complete  request  in  a 
category,  and  number  of  requests  in  a  category.  However,  practical 
limitations  of  the  data  collected  during  the  survey  resulted  in  a 
subset  of  the  27  categories  being  used,  where  time  to  complete  is  the 
release  duration,  and  the  nine  categories  of  type,  complexity  and 
priority  are  used  without  any  combinations. 
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c. -  Another  instance,  where  a  more  detailed  (rather  than  less 

detailed  as  in  the  baseline  profile  above)  capability  was  derived 
from  the  analysis,  is  the  software  supportability  risk  computation. 
The  simple  linear  conversion  model  should  be  replaced  by  the  more 
accurate  linear  regression  model,  or  the  even  better,  but  more 
computationally  intensive,  factor  regression  model. 

d.  Overall,  the  analysis  supported  the  current  risk  assessment 
methodology  reasonably  well  with  the  above  relatively  minor  modifi¬ 
cations. 

2.2.5  Conclusions/Recommendations . 

a.  During  this  study,  there  were  several  observations  about 

software  support  activities  that  merit  comment  and  recommendations 
for  additional  action  or  study.  These  observations  are  summarized  in 
table  2-2.  The  conclusions  from  the  data  analysis  are  summarized  in 
table  2-3. 

b.  These  observations  and  results  support  the  primary  objective 

of  this  study,  which  is  to  determine  whether  there  is  justification 
from  the  software  support  data  to  warrant  further  refinement  and 
application  of  a  risk  assessment  methodology.  The  authors  of  this 
report  conclude  that  justification  for  refinement  and  application 
does  exist.  Analysis  of  the  data  indicates  that  the  foundation  for 

the  methodology  has  been  created  and  that  the  methodology  is  now 

ready  for  refinement  and  actual  application  via  a  pilot  study. 
Refinement  is  still  required  in  order  to  complete  the  software  life 
cycle  evaluation  tool  and  adapt  the  other  evaluation  tools  to  conform 
to  the  RA  methodology.  The  following  ordered  steps  are  recommended: 

(1)  Adapt  the  software  maintainability  tool  to  measure 
against  the  user/supporter  baseline  agreement 
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Table  2-2. 
Summary  of  Problems 


PROBLEM 

1.  High  personnel  attrition  rate 

2.  Inconsistent  application  of  software 
configuration  management  system  (CMS) 

3.  Lack  of  agreement  on  definition  of 
terms 

4.  Modifications  to  software  take  too 
long  to  reach  the  field 


5.  Systems  support  apportioned  among 
multiple  ALCs 

6.  Contractors  not  transferring  the 
software  responsibility  to  support 
facilities 

7.  System  memory  and  processing 
constraints  for  embedded  (OFP) 
software  development 


SOLUTION 

Requires  further  study 

Set  and  enforce  standard 
CMS  procedures 

Terms  defined  as  part  of 
standard  CMS 

Study  modification,  testing, 
and  release  process  to 
streaml ine 

Requires  concept  change  for  ALC 
operation 

Need  more  information 


Technology  enhancement  and 
evolution  of  systems  over 
time.  Problem  recognition 
for  future  systems. 
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Table  2-3. 


Summary  of  Oata  Analysis  Conclusions 


ANALYSIS 

Software  Supportabi 1 ity  Risk 
Computation 


2.  Software  System  Release  Oata 


CONCLUSIONS 

la.  The  simple  linear  conversion  model 
is  not  supported. 

lb.  The  linear  regression  model  is 
supported  with  somewhat  low  con¬ 
fidence. 

lc.  The  factor  regression  model  is 
supported  with  most  confidence. 

The  four  driver  factors  are 
software  product,  software  support 
management,  software  personnel,  and 
software  support  systems. 

ld.  The  six  significant  factors  derived 
for  use  in  the  factor  regression  model 
coincide  with  elements  of  the  ' 

AFOTEC  software  supportabi 1 ity  • 

evaluation  hierarchy. 

2a.  The  person-months  per  change  is  not 
significantly  correlated  with  number 
of  source  lines,  proportion  of  cor¬ 
rection  requests,  proportion  of  low 
complexity  requests,  proportion  of 
normal  priority  requests,  percen¬ 
tage  of  high-level  language  source,  or 
average  skill  of  support  personnel. 

2b.  The  lack  of  any  strong  correlation 
in  the  results  of  2a  is  attributed 
largely  to  the  coarse  nature  of 
the  available  data. 


Historical  Maintenance  Profiles  3a.  There  was  not  strong  support  for 

stratifying  profile  data  by  site. 

3b.  There  was  some  support  for  strat¬ 
ifying  profile  data  by  software 
system  type,  although  the  1 ack  of 
data  for  some  types  hindered 
the  overall  conclusions. 
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(2)  Adapt  the  software  support  facility  evaluation  tool 
(ASSET)  to  measure  against  the  user/supporter  baseline 
agreement 

(3)  Complete  the  top  level  software  life  cycle  process 
evaluation  metrics  for  risk  assessment 

(4)  Apply  the  total  software  supportabi 1 ity  risk  assessment 
methodology  to  a  current  program.  Evaluate  results  of 
application  and  complete  technology  transfer  to  AFOTEC 
personnel 

(5)  Continue  the  collection  of  software  system  release  data. 
A  recommended  data  collection  form  and  procedure  for 
temporary  use  is  presented  in  section  3.6 

(6)  Develop  procedures  to  update  the  historical,  maintenance 
profiles  and  analysis  results  from  the  newly  collected 
software  system  release  data 

(7)  Continue  to  evolve  the  software  supportabi 1 ity  risk 
computation  regression  analysis  results.  Use  the  factor 
regression  model,  the  linear  regression  model,  and  the 
simple  linear  model  in  that  order  of  preference. 

(8)  Continue  to  analyze  potential  relationships  between 
person-months  per  change  and  other  system  level  variables 
such  as  percentage  of  low  complexity  change  requests. 
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SECTION  III 

OVERVIEW  OF  APPROACH 

3.1  INTRODUCTION. 

a.  From  the  very  beginning  of  this  study,  it  was  recognized  that 
a  standardized,  repeatable  method  was  necessary  to  collect  the 
software  maintenance  activity  data.  Meetings  were  held  with  AFOTEC 
personnel  shortly  after  initiation  of  this  subtask  to  gain  agreement 
on  the  types  of  data  to  be  collected  and  the  form/format  that  would 
be  used  to  collect  the  data.  Whereas  the  general  format  of  the 
survey  form  which  resulted  from  these  meetings  remained  constant 
during  the  study,  several  changes  occurred  during  the  course  of  the 
study  as  greater  insight  was  obtained  about  the  utility,  of  the  form 
and  the  types  of  data  desired.  This  report  will  not  discuss  the 
detailed  history  of  the  evaluation  of  the  form  from  its  first  to  its 
current  (seventh)  draft,  however,  discussion  of  the  pitfalls  of  some 
of  the  earlier  versions  will  be  discussed  in  section  3.3  of  this 
report. 

b.  In  conjunction  with  AFOTEC,  procedures  were  also  created  to 
set  up  and  perform  the  facility  visits.  These  procedures  were  modi¬ 
fied  as  the  study  progressed  and  more  was  learned  about  what 
approaches  worked  best. 

c.  It  would  be  wrong  to  assume  that  using  the  survey  form  or 
facility  visit  procedures  discussed  in  this  report  will  guarantee 
complete  success  in  collecting  desired  information.  The  uniqueness 
of  the  facilities  and  svstems  visited  meant  that  constant  minor 
adjustments  were  being  made  to  respond  to  each  situation.  This  state 
of  affairs  will  likely  continue  until  a  more  uniform  system  for 
collecting  and  reporting  software  maintenance  activity  data  is 
enforced  Air  Force-  or  DoD-wide. 


1 1 1 -I 


Vv'.VXV 


/  v  W 


THE  BOM  CORPORATION 


BDM/A-S5-0510-TR 


d.  As  an  initial  effort,  recommendations  are  made  in  section  3.6 

for  a  temporary  data  collection  form  to  be  used  by  all  support  sites. 
This  data  would  be  necessary  for  the  upgrade  and  evolution  of  the 
data  collected  during  this  study  in  support  of  the  software 
supportability  risk  evaluation. 

3.2  ASSUMPTIONS  OF  SOFTWARE  MAINTENANCE  ACTIVITY. 

a.  The  "view"  of  the  software  support  activity  evolved  somewhat 
during  the  process  of  integrating  the  collected  data  in  order  to 
achieve  consistency  and  commonality  across  software  systems.  The 

resulting  assumptions  of  the  activity  which  most  systems  follow  to 
some  extent  have  these  major  properties: 

(1)  Responsibi 1 ity.  Software  support  has  been  assumed  either 

formally  through  a  Program  Management  Responsibility 

Transfer  (PMRT),  or  informally  such  as  after  Initial 

Operational  Capability  (IOC).  Changes  are  being 
processed  to  an  operational  system  within  a  formal 
configuration  management  process. 

(2)  81ock  Release.  The  software  maintenance  process  is  based 
on  production  -f  a  scheduled  or  unscheduled  block 
release.  Each  change  request  is  a  (reasonably)  formal 
item  which  is  tracked  under  configuration  management 
status  accounting.  Releases  may  overlap  and  share 
resources.  The  term  "percentage  of  resources  dedicated 
to  a  release"  is  used  to  balance  the  available  resources 
which  account  for  the  productivity  of  a  given  release. 

(3)  Change  Request.  A  software  change  request  can  be  classi¬ 
fied  by  type,  complexity,  and  priority.  The  type  is 

correction,  enhancement,  or  conversion.  The  complexity 
is  low,  medium,  or  high.  The  priority  is  normal,  urgent, 
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or  emergency.  Software  change  requests  are  prioritized 
and  grouped  into  a  change  block  or  release.  The  change 
block  is  processed  as  a  software  development  effort, 
including  analysis,  requirements,  design,  code  and  unit 
test,  integration  and  operational  test,  and  formal  deli¬ 
very  and  installation.  The  emphasis  on  each  phase  will 
depend  upon  the  change  block  and  the  operational  system. 

(4)  Data.  Major  software  maintenance  parameters  of  interest 
include: 

a)  Block  release  start  date,  engineering  end  date,  field 
date 

b)  Number  of  personnel  assigned  to  block  release  and 
percentage  of  the  time  these  personnel  are  dedicated 
to  the  release 

c)  Skill  level  of  personnel  assigned  to  block  release 

d)  Estimated  level  of  resource  requirements  (personnel 
and  systems)  for  block  release  at  start  date 

e)  Actual  level  of  resources  consumed  (personnel  and 
systems)  for  block  release  at  engineering  end  date 

f)  For  each,  change  request  in  the  block  release,  the 
type,  complexity,  priority,  estimated  and  actual 
resource  requirements,  configuration  control  dates 

g)  The  total  number  of  change  requests  which  were 
carried  over  for  consideration  in  a  future  block 
release. 
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(5)  Personnel .  Resources  include  organic  (military/civilian 
government)  and/or  contractor  personnel.  Personnel  are 
those  directly  associated  with  the  given  software  system 
in  management,  technical,  support,  or  contractor  capa¬ 
city.  Resources  may  be  shared  across  software  systems. 
The  term  "percentage  of  resources  dedicated  to  the 
software  system"  is  used  to  balance  the  available 
resources  which  account  for  the  productivity  of  a  given 
release. 

b.  A  software  system  may  not  satisfy  the  above  assumptions  in  all 
aspects.  The  majority  of  the  software  systems  for  which  data  was 
obtained  did  satisfy  many  of  these  assumptions.  Problems,  excep¬ 
tions,  workarounds,  and  so  forth  will  be  discussed,  as  appropriate, 
throughout  this  report. 

3.3  SURVEY  FORMAT. 

a.  The  Data  Survey  Form  (Oraft  7,  dated  29  April  1985)  is 
presented  in  appendix  C  of  this  report.  The  basic  sections  of  this 
form  will  be  discussed  in  the  following  paragraphs. 

b.  The  types  of  data  collected  using  the  form  can  be  divided  into 
three  categories:  (1)  background  data  on  each  software  system 
surveyed;  (2)  a  high-level,  subjective  assessment  of  the  adequacy  of 
the  software  support  being  provided  for  each  major  software  system; 
and  (3)  actual-  software  maintenance  data  for  each  major  software 
system.  Hence  the  data  collection  form  is  also  divided  into  these 
categories.  The  data  collected  in  each  category  will  be  discussed  in 
some  detai 1 . 

3.3.1  Survey  Form  -  Preface  Information. 


a.  Information  was  presented  at  the  front  of  the  form  as  an 
introduction  to  the  purpose  of  the  requested  software  support  data. 
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and  three  pages  of  definitions  were  attached.  This  format  was 
selected  because  it  appeared  advantageous  to  mail  a  copy  of  the 
survey  form  to  the  individuals  from  whom  desired  information  was 
anticipated  prior  to  the  facility  visit  being  conducted.  The  intent 
was  to  minimize  the  time  spent  during  the  facility  visits  introducing 
the  purpose  of  the  survey,  thereby  concentrating  on  obtaining  all 
applicable  data.  This  approach  had  some  measure  of  success,  because 
early  drafts  of  the  survey  form  did  not  have  an  introduction.  It  was 
discovered  that  most  recipients  of  the  survey  form  at  that  time 
waited  until  the  facility  visit  was  being  conducted  before 
researching  the  requested  information.  During  later  facility  visits, 
where  the  preface  information  was  provided  with  the  survey  form, 
there  was  a  greater  tendency  for  individuals  to  have  some  of  the 
desired  information  ready  when  the  facility  visit  began. 

b.  It  was  also  evident  from  early  versions  of  the  survey  form 
that  there  was  potential  for  disagreement  or  misunderstanding  about 
the  meaning  of  some  of  the  terms  used  in  the  form  itself.  Therefore, 
to  prevent  such  occurrences,  and  to  provide  an  increased  potential 
for  obtaining  consistent  responses  among  facility  visits,  definitions 
of  key  terms  were  included  in  the  preface  to  the  survey.  These  defi¬ 
nitions  have  been  used  consistently  throughout  the  risk  assessment 
study,  including  the  feasibility  study  for  the  RAMSS. 

3.3.2  Survey  Form  -  Software  Background  Data. 

a.  The  background  data  requested  consists  of  two  basic  types: 
(1)  identification  data  (system  name,  software  system  name,  software 
system  type)  and  (2)  description  data,  which  was  necessary  to  better 
understand  the  characteristics  of  the  software  support  being 
provided. 

b.  Not  all  of  the  description  data  gathered  was  intended  to  be 
useful  in  developing  risk  profiles  of  maintenance  activities. 
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However,  given  the  opportunity  to  collect  this  information,  it  was 
decided  the  data  might  be  used  for  future  additional  analysis.  For 
instance,  the  size  of  the  software  system  (#  CSCIs,  #  Modules, 
#  Source  lines)  and  the  languages  (and  percent  use)  information  were 
not  planned  to  be  part  of  the  risk  assessment  methodology.  Analysis 
(see  section  IV)  indicates  this  information  was  not  significant  for 
the  data  reported.  However,  given  enough  data  more  accurate  than  was 
collected,  the  significance  of  this  information  might  change.  It 
might  be  interesting,  given  that  one  knows  the  size  and  language  of  a 
system  being  evaluated  by  the  RAMSS,  to  assess  the  risk  against 
systems  whose  size  and  language  are  close  to  that  of  the  target 
system. 


c.  The  description  data  which  are  basic  to  the  risk  profile 
development  are  the  number  of  personnel  currently  supporting  the 
software  system  and  a  measure  of  the  amount  of  time  such  personnel 
are  dedicated  to  supporting  the  system.  This  information  must  be 
obtained  as  accurately  as  possible  because  the  model  depends  upon  a 
measure  of  the  historical  productivity  for  the  software  system 
support.  Other  information  on  personnel  skill  levels  could  be  very 
important,  however  analysis  did  not  seem  to  provide  any  consistent 
results  as  to  how  personnel  skill  levels  affect  maintenance  effort. 
Still,  with  more  accurate  data  than  was  collected,  such  information 
might  be  an  indication  of  why  some  productivity  measures  are 
different. 

d.  The  remainder  of  the  data  collected  under  background  informa¬ 
tion  primarily  indicates  characteristics  of  software  support 
activities  that  represent  problems,  or  special  circumstances  which 
might  explain  why  data  collected  on  some  systems  are  significantly 
different  from  what  was  expected  (for  example,  a  temporary  freeze  on 
enhancement,  or  a  high  attrition  rate  of  experienced  personnel). 


THE  BOM  CORPORATION 


BDM/A- 85-05 10-TR 


3.3.3  Survey  Form  -  Software  Assessment  Data. 

a.  The  purpose  for  this  part  of  the  survey  form  was  to  collect 
information  which  allowed  a  high  level  correlation  between  the  AFOTEC 
supportability  evaluation  tools  (which  are  intended  to  "measure"  the 
risk  of  supporting  software)  and  a  subjective  assessment  of  risk  by 
personnel  who  are  currently  supporting  the  software.  This  correla¬ 
tion  was  intended  to  provide  information  about  how  close,  across 
systems,  the  proposed  evaluation  tools  come  to  measuring  perceived 
risk.  This  information  was  influential  in  determining  the  relation¬ 
ship  between  numeric  evaluation  scores  obtained  by  applying  the 
evaluation  tools  and  subsequent  conversion  to  a  risk  metric.  For 
example,  the  initial  attempt  to  relate  a  numeric  evaluation  score  of 
software  system  supportability  (on  a  1  to  6  scale,  where  1  is  worst 
and  6  is  best)  to  a  risk  assessment  score  (on  a  scale  of  0  to  1, 
where  0  is  absolute  certainty  of  being  able  to  support  the  software 
system  and  1  is  absolute  certainty  of  not  being  able  to  support  the 
software  system)  might  indicate  that  a  linear  relationship  exists. 
In  other  words  if  M  =  overall  support  score  (1-6),  and  R  s  risk 
(0-1),  then  linearly: 


Analyzing  (see  section  IV)  the  software  assessment  data  collected  in 
this  section  of  the  survey  form  indicates  a  different  (nonlinear) 
relationship  is  more  likely  to  be  valid. 

b.  The  survey  form  requests  ratings  for  the  attributes  used  to 
measure  software  supportability  at  both  delivery  and  current  times. 
Delivery  time  means  that  time  when  the  software  system  was  turned 
over  to  the  supporter  for  maintenance.  This  information  was  desired 
to  indicate  trends  in  the  overall  support  of  software  following 
delivery.  It  could  be  used  to  pinpoint  problem  areas  if  the  scores 
in  various  areas  went  from  positive  values  at  delivery  time  to 
negative  values  at  the  current  time. 
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c.  Several  comments  are  appropriate  about  the  evolution  of  this 
section  of  the  form.  The  survey  form  originally  requested  an  evalu¬ 
ation  on  a  scale  of  1  to  100  (worst  to  best)  of  the  software  support 
system's  attributes  as  described  by  the  tools  used  to  measure  the 
maintainability  of  the  software  system.  Later  in  this  study,  the 
scale  was  changed  from  "1  to  100"  to  "-50  to  +50,"  with  a  condition 
that  "zero"  not  be  chosen.  This  change  was  made  because  it  seemed 
easier  to  associate  a  negative  number  with  a  negative  (or  inadequate) 
rating.  Zero  was  eliminated  as  a  choice  because  its  absence  forced 
raters  to  chose  between  whether  the  support  attribute  was  either 
adequate  or  inadequate  for  the  software  support  system.  Also,  this 
forced  the  raters  to  think  harder  about  what  a  term  meant  and  to  get 
more  information  if  not  enough  was  present  to  make  a  decision.  If 
the  rater  isn't  sure  what  to  put,  zero  may  appear  to  be  an  easy  out. 


d.  Other  minor  changes  to  this  section  of  the  survey  form 
included:  (1)  an  expansion  of  information  in  the  software  support 
environment  section  to  obtain  more  data  on  the  adequacy  of  software 
support  systems,  and  (2)  the  addition  of  an  overall  evaluation  (scale 
of  -50  to  +50)  for  the  supportability  of  the  entire  system.  This 
latter  addition  provides  a  three  level  correlation  of  assessment 
data.  For  example,  an  overall  score  of  +45  would  not  correlate  to 
sub-scores  of  negative  values.  Should  this  happen,  it  might  indicate 
that  the  rater  took  something  into  account  that  the  evaluation  metho¬ 
dology  does  not  measure.  This  information  could  be  valuable  for 
evaluation  methodology  enhancement. 

3.3.4  Survey  Form  -  Desirable  Maintenance  Data. 

a.  This  section  of  the  survey  form  lists  the  major  data  items 
which  were  desirable  to  be  collected.  It  probably  was  the  most 
changed  portion  of  the  original  form.  These  changes  occurred  because 
initial  attempts  at  collecting  the  data  at  the  level  of  detail 
proposed  in  the  RAMSS  report  (reference  8.3)  were  too  time  consuming. 
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The  data  proposed  for  collection  in  reference  8.3  included:  for  each 
software  change  or  maintenance  activity  (such  activity  being  docu¬ 
mented  by  a  software  change  request,  software  trouble  report, 
software  problem  report,  software  maintenance  request,  or  other 
traceable  entity),  the  average  time  to  complete  each  priority 
(emergency,  urgent,  or  normal)  activity  (enhancement,  conversion,  or 
correction)  with  a  high,  medium  or  low  complexity.  More  simply 
stated,  the  model  required  that  information  on  each  software  mainte¬ 
nance  activity  be  classified  in  one  of  27  categories 
(3  priorities  x  3  types  x  3  complexity  levels)  and  that  the  precise 
time  to  perform  such  activity  be  known. 

b.  Unfortunately,  the  information  was  not  usually  present  in  the 
facilities  visited  at  the  above  level  of  detail.  In  particular,  the 
information  about  the  performance  time  for  each  maintenance  activity 
was  not  being  accurately  recorded  at  the  facilities.  In  addition, 
while  it  was  usually  possible  to  indicate  how  many  total  maintenance 
activities  were  emergency,  urgent,  or  normal  for  each  system,  it  was 
not  usually  possible  for  example,  to  break  the  number  of  emergency 
changes  into  high,  medium,  or  low  complexity. 

c.  The  information  requested  has  evolved  into  a  more  reasonable, 
and  by  experience  more  collectable,  request  for  software  maintenance 
activity  by  block  release.  A  block  release  is  a  defined  collection 
of  maintenance  activities  which  form  an  operational  baseline  for  the 
system.  Information  was  requested  about  each  block  release,  such  as 
specific  software  changes  implemented,  estimated  person  effort, 
actual  person  effort,  engineering  start  and  end  dates,  and  times  from 
engineering  end  date  until  release  was  fielded.  Information  on  each 
change  was  still  requested  as  before  (put  in  one  of  27  categories) 
because,  were  it  available,  much  better  evaluations  of  risk  could  be 
performed  by  being  able  to  identify  the  drivers  of  risk  at  a  lower 
level.  Finally,  additional  data  was  requested  about  such  items  as 
the  number  of  software  change  requests  carried  over  from  the  previous 
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year,  opened  during  the  current  year,  and  closed  during  the  current 
year.  This  data  was  not  generally  available,  but  would  provide 
information  on  the  ability  of  the  support  facility  to  keep  up  with 
the  support  requests  and  should  correlate  with  other  risk  assessment 
variables. 

3.4  FACILITY  VISIT  PROCEDURES. 

While  there  were  no  formal  facility  visit  procedures  estab¬ 
lished  at  the  beginning  of  this  study,  a  discussion  of  the  general 
procedures  followed  will  hopefully  be  of  benefit  to  future  study 
efforts.  The  procedures  can  be  divided  into  pre-visit,  on-site 
visit,  and  post-visit  activities.  Each  of  these  activities  has 
proven  to  be  necessary  for  a  successful  data  collection  effort. 

3.4.1  Pre-Visit  Procedures. 

a.  Pre-visit  procedures  and  activities  were  primarily  accom¬ 

plished  by  AFOTEC.  They  basically  consisted  of  telephoning  the 
facilities  targeted  for  visits  several  weeks  in  advance  of  the  visit 

to  confirm  what  software  systems  were  being  supported  and  who  would 

be  the  primary  points  of  contact.  AFOTEC  personnel  also  coordinated 
an  appropriate  time  for  the  visit,  and  briefly  explained  over  the 
telephone  the  purpose  of  the  visit.  AFOTEC  designated  appropriate 
government  personnel  to  accompany  BDM  personnel  on  the  visits.  In 

addition,  AFOTEC  sent  a  letter  to  each  office  contacted  (see 

figure  3-1  for  a  sample  letter).  The  letter  verified  the  visit,  its 
purpose,  and  contained  a  copy  of  the  survey  form  as  an  attachment. 
When  possible,  BOM  and  AFOTEC  personnel  reviewed  any  available  docu¬ 
mentation  on  the  systems  to  be  visited  prior  to  the  trip. 

b.  There  are  three  key  items  which  should  be  noted  about  pre¬ 
visit  activities: 
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DEPARTMENT  OF  THE  AIR  FORCE 

HEAOOUARTERS  AIR  FORCE  OPERATIONAL  TEST  AND  EVALUATION  CENTER 
KIRTLANO  AIR  FORCE  BASE.  NEW  MEXICO  87117  5000 


Request  for  Visit  and  Site  Survey 


to  (Two  or  Three  Letter  Office,  as  appropriate) 


1.  Reference  telecons  between  Lt  Col  Smith  and  Capt  Tomlin  on 

2  July  85,  and,  between  Mr  Jones  and  Caot  Tomlin,  and  Mai  Hunter 
and  Capt  Tomlin  on  5  July  85. 

2.  We  would  like  to  visit  your  facilities  for  two  days  beginning 
23  July  85  to  collect  a  variety  of  data  described  in  the  attached 
Site  Survey  Form.  The  survey  is  an  Important  part  of  our  efforts 
to  develop  a  methodology  to  assess  the  risk  of  supporting  system 
software.  This  assessment  is  to  be  used  within  the  initial  oper¬ 
ational  test  environment.  Tour  valuable  assistance  and  coopera¬ 
tion  will  provide  a  significant  contribution  to  this  effort. 

3.  We  have  initially  developed  a  methodology  of  assessing  soft¬ 
ware  support  risk.  However,  the  methodology  depends  on  develop¬ 
ing  profiles  of  software  support  activities.  The  attached  site 
survey  has  been  designed  to  capture  the  data  needed  to  build  the 
profiles  of  activity  and  the  background  information  necessary  to 
describe  historical  and  current  experiences.  We  have  found  it 
essential  to  key  in  on  the  senior  software  engineer  (or  equiv¬ 
alent)  f-or  each  system  as  our  starting  point  and  source  of  the 
majority  of  Information.  We  have  also  found  copies  of  the  CRISP 
and  0/ S  CMP  for  each  system  most  helpful.  We  are  supplying  the 
survey  in  advance  so  that  each  senior  software  engineer  may  be 
aware  of  the  type  of  Information  we  are  seeking,  determine  if 
other  personnel  should  be  interviewed,  and  prepare  data  sources 
as  appropriate  . 

4.  This  entire  effort  to  develop  a  risk  assessment  of  software 

s uppo r t a b 1 1 1 t y  has  been  with  the  aid  of  the  BDM  Corporation.  Two 
of  their  personnel  and  one  from  our  Software  Evaluation  Division 
will  make  the  visit.  Upon  your  confirmation,  we  will  send  a 
message  with  names  and  security  clearences  the  week  prior  to  our 
visit . 

5.  Questions  and  comments  should  be  addressed  to  Capt  Eric 
Tomlin,  HO  AFOTEC/ LG5T ,  AV  246-1381,  or,  Maj  Gary  Horlbeck,  HO 
AF0TEC/LG5T,  AV  246-1254. 

FOR  THE  COMMANDER 


(Signature  Block) 


1  Atch 

Site  Survev  Package 


Figure  3-1.  Sample  Letter 
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(1)  The  success  of  a  facility  visit  was  highly  dependent  upon 
AFOTEC's  designation  of  a  "key"  government  (AFOTEC) 
employee  to  accompany  BOM  personnel  on  the  visit.  This 
"key"  person  has  usually  been  previously  assigned  to  the 
facility  or  its  vicinity,  and/or  was  knowledgeable  of  the 
personnel  who  were  visited.  This  designation  occurred  in 
all  visits  except  one. 

(2)  The  earlier  letters  sent  to  prospective  facilities  did 
not  request  that  survey  form  information  be  prepared  in 
advance.  Later  versions  of  the  letters  requested  that, 
when  possible,  the  survey  form  information  should  be 
available  at  the  time  the  visit  was  performed.  This 
proved  to  be  beneficial  in  some  instances.  Most  indivi¬ 
duals,  however,  waited  unMl  the  facility  visit  was 
performed  before  attempting  to  complete  the  survey  form. 

(3)  A  final  key  item  was  the  early  identification  of  the 
senior  software  engineer  (or  equivalent)  on  each  system 
visited.  The  knowledge  of  this  individual,  who  almost 
always  had  been  working  with  the  system  for  some  time  and 
was  therefore  very  familiar  with  the  history  of  the 
system  as  well  as  its  current  status,  has  proven  to  be 
extremely  important  to  obtaining  the  necessary  data  in  a 
timely  manner. 

3.4.2  On-Site  Visit  Procedures. 

a.  As  with  the  pre-visit  procedures,  it  was  learned  that  some 
methods  of  approach  worked  better  than  others  when  the  facility  visit 
was  actually  being  performed.'  Because  of  the  time  constraint 
involved  in  the  visits,  there  was  a  natural  tendency  to  want  to  get 
right  to  the  software  engineers  to  start  collecting  information. 
However,  in  all  cases  some  time  was  spent  with  upper  level  management 
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explaining  the  purpose  of  the  visit.  This  was  a  necessary  step  that 
should  not  be  avoided  because  it  enhanced  cooperation  during  the 
visit. 

b.  The  next  logical  step  was  to  get  together  with  as  many  of  the 
software  engineers  as  possible  to  explain  the  purpose  of  the  visit 
and  discuss  the  information  desired.  It  was  not  possible  to  perform 
this  step  for  a  large  group  because  the  software  engineers  were 
spread  out  in  various  locations  at  the  facilities.  For  the  most 
part,  separate  meetings  with  each  system  or  subsystem  software 
engineers  were  performed.  The  methods  which  appeared  to  provide  the 
most  timely  information  included: 

(1)  AFOTEC  personnel  gave  a  short  introduction  to  the 
briefing  purpose. 

(2)  BOM  personnel  presented  about  a  30-  to  45-minute 
discussion  of  the  methodology  and  data  desired. 

(3)  At  the  conclusion  of  the  meeting,  decisions  were  made 
about  when  to  return  to  the  software  engineer  to  collect 
the  data  and  answer  questions. 

c,  A  key  piece  of  advice:  make  every  attempt  to  collect  all 
appropriate  data,  documents,  evaluation  forms,  or  other  important 
information  before  concluding  the  visit.  There  are  several 
instances,  when  information  which  was  promised  has  not  been  received. 


d.  Other  areas  to  be  aware  of  during  the  data  collection  process 
i nc 1 ude : 

(1)  Instruct  the  software  engineers  on  the  following  items 
regarding  the  software  assessment  data  form  in  the 
survey.  All  blanks  are  to  be  completed  on  this  form 


1 1 1-13 


.‘v%V 


1  »  ■  ~  -  '  *  *  “  *  *  »  -**••*•<  w  „  «  m  m  _  m  k 


* 


THE  BOM  CORPORATION 


BDM/A-85-0510-TR 


(only  a  few  will  have  non-appl icable  items  under  software 
support  environment).  Do  not  enter  a  zero.  If  program 
management  responsibility  for  the  system  was  never 
transferred  (many  systems  were  developed  organically),  or 
"At  Delivery"  has  no  meaning,  then  only  complete  the 
"Current"  column. 

(2)  Check  over  the  completed  form  for  completeness  and  con¬ 
sistency.  Several  individuals  were  confused  about  the 

assignment  of  supportabi 1 ity  risk. 

(3)  Ensure  that  all  data  received  is  understood  with  regard 
to  its  meaning  and  content.  Look  for  completeness  and 
consistency  in  the  software  maintenance  data.  The  time 
to  clear  up  problems  is  during  the  visit,  not  afterward. 

3.4.3  Post-Visit  Procedures.  SOM  personnel  have  compiled  notebooks 
from  each  visit.  These  notebooks  contain  a  copy  of  the  trip  reports, 
and  an  indexed  filing  of  all  documents,  notes,  organization  charts, 
software  evaluation  forms,  and  maintenance  data  collected  during  the 
facility  visit.  These  notebooks  will  be  maintained  by  BDM  personnel. 

3.5  FACILITIES  VISITED  AND  SYSTEMS  EXAMINED. 

a.  A  total  of  six  data  collection  visits  were  performed  during 
this  study.  The  facilities  and  dates  of  the  trips  are  summarized  in 
table  3-1. 

b.  Summaries  of  each  trip  may  be  found  in  appendix  F  of  this 
document. 
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Table  3-1. 
Trip  Schedule 


NORAD 

Warner  Robins  ALC 
Sacramento  ALC 
Castle  AFB 
Ogden  ALC 

♦Oklahoma  City  ALC/Tinker  AFB 
Langley  AFB 


DATE  VISITED 

7-9  January  1985 

28  January  -  1  February  1985 

24-26  February  1985 

27-28  February  1985 

22-25  April  1985 

13-16  May  1985 

21-25  July  1985 


♦The  E-3A  facilities  at  Tinker  AFB  were  not  associated  with  the 
Oklahoma  City  ALC. 

c.  Table  3-2  lists,  by  facility  visited,  the  systems/subsystems 
examined  to  date  by  this  study.  The  following  abbreviations  apply: 


(1)  NORAD 


North  American  Aerospace  Defense  Command, 
Cheyenne  Mountain  Complex,  Colorado 
Springs,  Colorado 


(2)  WR-ALC 


Warner  Robins  ALC,  Robins  AFB,  Georgia 


(3)  SM-ALC 


Sacramento  ALC,  McClellan  AFB,  California 


(4)  CASTLE  AFB  Castle  AFB,  California 


(5)  00-ALC 


Ogden  ALC,  Hill  AFB,  Utah 


(6)  OC-ALC 


Oklahoma  City  ALC,  Tinker  AFB,  Oklahoma 


(7)  TINKER 


E-3A  Facility,  Tinker  AFB,  Oklahoma 


(3)  LANGLEY 


Langley  AFB,  Virginia 
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Table  3-2. 


Software  Systems  Examined 


SITE 

SYSTEM 

SOFTWARE  SYSTEM 

NORAD 

CSS 

CSS 

NORAD 

MDS 

MDS 

NORAD 

MEBU 

MEBU 

NORAD 

NCS 

NCS 

NORAD 

SSC 

SSC 

WR-ALC 

ALR-46 

ALR-46 

WR-ALC 

ALR-69 

ALR-69 

WR-ALC 

AN/ALQ- 131 

AGEOP 

WR-ALC 

AN/ALQ-131 

BTG 

WR-ALC 

AN/ALQ- 131 

OFP 

WR-ALC 

ALQ-131 

UUT 

WR-ALC 

APR-38 

APR-38 

WR-ALC 

B- 52  EVS  ATE 

ASQ-151 

WR-ALC 

E-3A  AVIONICS  ATE 

AN/GSM-285(B) 

WR-ALC 

E-3A  AVIONICS  ATE 

AN/GSM-285(W) 

WR-ALC 

F-15 

CC 

WR-ALC 

F- 15 

RADAR 

WR-ALC 

F-15  AVIONICS  ATE 

ADTS.AIS 

WR-ALC 

JTIDS 

ASIT/OCP 

WR-ALC 

JT  IDS 

E-3A  AWACS/OCP 

WR-ALC 

JTIDS 

SP/USER 

WR-ALC 

JTIDS 

SYS  EXERCISER 

WR-ALC 

PAVE  TACK 

AISF 

WR-ALC 

PAVE  TACK 

OFP 

SM-ALC 

F-111D 

WEAP-NAV  COMPUTER 

SM-ALC 

F-111F 

WEAP-NAV  COMPUTER 

SM-ALC 

FB-111A 

WEAP-NAV  COMPUTER 

CASTLE  AFB 

A  T-4 

A  T-4  SIMULATOR 

CASTLE  AF8 

B-52 

CPT 

CASTLE  AFB 

B  -  5  2 

WST 

CASTLE  AFB 

KC-135 

WST 

OO-ALC 

F  - 16 

FCC 

OO-ALC 

F  - 16 

HUO 

OO-ALC 

F  - 16 

OFT 

OO-ALC 

F  - 16 

FCR 

OO-ALC 

F- 16 

SMS 

OO-ALC 

F-4 

MOTS 

OO-ALC 

F-4E 

AN/ARN- 101 

OC-ALC 

F-4G 

AN/ARN- 101 

OO-ALC 

F-4G 

LRU-l/ACM 

OO-ALC 

MINUTEMAN 

WING  11/2015 

OO-ALC 

minuteman 

WING  VI/HS-29 

OO-ALC 

MINUTEMAN 

WINGS/HS-28 

ni-16 
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Table  3-2 


Software 

Systems  Examined 

(Continued) 

00-ALC 

MINUTEMAN 

II 

SSAS/CAPS 

00-ALC 

MINUTEMAN 

II 

WING  V/HEG/RATS 

00-ALC 

MINUTEMAN 

II 

WING  VI/HEG/RATS 

00-ALC 

RF-4 

CAN/ARN-101 

OC-ALC 

ALCM 

LEVEL  I  TEST 

OC-ALC 

ALCM 

LOADED  PYLON  TEST 

OC-ALC 

ALCM 

OFP 

OC-ALC 

B-1B 

CADC 

OC-ALC 

8- IB 

CITS 

OC-ALC 

B-1B 

EMUX 

OC-ALC 

B-IB 

F/CGMS 

OC-ALC 

8-  IB 

INS 

OC-ALC 

B-IB 

ORS 

OC-AlC 

8-52 

8NST 

OC-ALC 

B- 52 

FTSS 

OC-ALC 

B- 52 

MC-1  EXEC 

00-ALC 

8-52 

MC-2  EXEC 

OC-ALC 

E-3 

INS 

OC-ALC 

E-3A 

OMEGA 

OC-ALC 

E-3A 

SMCP 

OC-ALC 

E-3A 

SRCP 

OC-ALC 

E-3A 

SRGSCP 

OC-ALC 

GLCM 

DPS 

OC-ALC 

GLCM 

M-DTD 

OC-ALC 

GLCM 

MPT 

OC-ALC 

GLCM 

OFP 

OL-ALC 

GLCM 

WCS 

OC-ALC 

SRAM 

OFP 

TINKER 

E-3A 

AOCP 

TINKER 

E-3A 

UTIL  SUPP  S/W 

LANGLEY 

JTIOS 

ASIT/TPOCP 

LANGLEY 

STRTS 

STRTS 

LANGLEY 

TACS 

6AFMS 

LANGLEY 

TIPI 

DC/SR 

LANGLEY 

TIPI 

I I/MARRES/TEREC 

LANGLEY 

407L 

HUGHES  UTIL 

LANGLEY 

40  7  L 

IBM  UTIL 

LANGLEY 

40  7  L 

IORP/IMPP 
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3.6  RECOMMENOED  SITE  DATA  COLLECTION  FORM. 

On  the  basis  of  the  experience  and  analyses  from  this  study, 
it  is  recommended  that  data  be  collected  for  each  software  system 
release  at  each  software  support  site  for  each  software  system  being 
supported.  This  subsection  includes  a  definition  of  the  data  items 
to  be  collected,  and  recommended  temporary  procedures  for  getting  the 
data  integrated  into  the  AFOTEC  Software  Maintenance  Profiles.  Tools 
to  automate  the  data  integration  would  need  to  be  constructed. 

3.6.1  Data  Items.  The  recommended  data  items  to  be  collected  for 
each  software  system  release  are  shown  in  figure  3-2.  Each  of  these 
data  items  is  briefly  discussed  in  the  following  paragraphs. 
Although  many  other  data  items  are  of  interest  (such  as  estimated 
versus  actual  values),  the  combination  of  the  analysis  results 
(section  IV)  and  practical  concern  for  site  personnel  time 
constraints  limited  the  data  collection  effort  to  the  specified 
items.  Experience  from  the  numerous  site  visits  indicates  that  the 
recommended  data  items  are  available  during  the  software  release 
effort  and  would  not  be  time  consuming  to  collect  and  record  on  a 
compact  form  such  as  shown  in  figure  3-2. 

3.6. 1.1  SITE.  The  SITE  is  the  named  location  which  provides  the 
organic  management  and/or  technical  support  for  the  subject  software 
system. 

Example:  Ogden  Air  Logistics  Center  (00-ALC) 

3. 6. 1.2  SYSTEM.  The  SYSTEM  is  the  collective  set  of  hardware/ 
software  of  which  the  subject  software  system  is  a  part. 

Example:  F - 16 


3. 6. 1.3  SOFTWARE  SYSTEM. 


The  SOFTWARE  SYSTEM  is  the  name  of  the 


collective  software  package  (documentation,  source,  object,  command 
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1.  site 


5.  SIZE  (K  LINES) 


COMPUTER  SOFTWARE  RELEASE  DATA  RECORD 


3.  SOFTWARE  SYSTEM 


2.  SYSTEM 


6.  LANGUAGE 


%  SOURCE  LINES 
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OATE 


4.  SOFTWARE  SYSTEM  TYPE 


HOL.(Y/N 


7.  PERSONNEL 

(LOWEST) 

SKILL  LEVEL  1. 

SKILL  LEVEL  2. 

SKILL  LEVEL  3. 

SKILL  LEVEL  4. 

(HIGHEST) 

SKILL  LEVEL  5. 

#  ORGANIC 


%  DEDICATED 


#  CONTRACTOR  %  DEDICATED 


12.  RELEASE  CHANGE  DATA  (USE  ADDITIONAL  ATTACHMENTS  AS  NECESSARY) 


CHANGE  REQUEST  OPEN 
_ ID _  DATE 


CLOSE  ACTUAL  TYPE  COMPLEXITY  PRIORITY 

OATE  PERSON  MONTHS  (C.E.V)  (L.M.H)  (N.U.E) 


AFOTEC  FORM  XXXX 
SEP  85 


as  0510 n  III  01 


Figure  3-2.  Recommended  Software  Release  Oata 
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8.  RELEASE  10/VERSION 

9.  RELEASE  START  OATE 

10.  RELEASE  ENGINEERING 

11.  RELEASE  FIELD  OATE 

COMPLETION  DATE 
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language)  for  which  change  release  data  is  being  reported.  The 
software  may  be  (but  need  not  be)  at  the  CSCI  level. 

Example:  Stores  Management  System  (SMS) 

3. 6. 1.4  SOFTWARE  SYSTEM  TYPE.  The  SOFTWARE  SYSTEM  TYPE  is  one  of 
the  following  categories: 

OFP  -  Operational  Flight  Program 
C-E  -  Communication-Electronics 
EW  -  Electronic  Warfare 

ATD  -  Aircrew  Training  Device/Operational  Flight  Trainer 
ATE  -  Automatic  Test  Equipment 
SIM  -  Simulation 

SUP  -  Support  (system  or  application) 

Example:  OFP 

3. 6. 1.5  SIZE.  The  SIZE  is  the  number  of  computer  language  source 
lines  in  thousands  (K)  for  the  subject  software  system.  Percentage 
of  comments  should  be  estimated. 

Example:  153K  (25%  Comments) 

3.6. 1.6  LANGUAGE.  The  LANGUAGE  is  a  list  of  computer  programming 
and  command/test  languages  in  which  the  subject  software  system 
source  is  written.  Language  includes  usual  programming  languages 
such  as  FORTRAN,  JOVIAL,  COBOL,  Ada,  Assembly,  etc.  The  percentage 
of  each  language's  source  lines  should  be  specified  along  with  an 
indication  (Y=yes,  N=no)  whether  the  language  is  a  High  Order 
Language  (HOL)  or  not. 

Example:  8080  Assembly  100%  N 

3. 6. 1.7  PERSONNEL. 

a.  The  PERSONNEL  is  the  count  of  all  persons  assigned  the 
direct  support  responsibility  for  the  subject  software  system. 
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Functionally,  these  personnel  represent  managers,  maintainers, 
programmers,  analysts,  configuration/quality  assurance  personnel, 
testers,  etc.  The  personnel  counts  should  be  separated  into  organic 
(civilian  and  Air  Force)  and  contractor,  and  also  by  approximate 
skill  level. 

Skill  level  1:  the  least  experienced  and  knowledgeable  personnel. 

Skill  level  5:  the  most  experienced  and  knowledgeable  personnel. 

Skill  level  2,  3,  4:  the  ranges  between  the  levels  1  and  5. 

Typically,  the  junior  personnel  with  little  experience  or  those  per¬ 
sonnel  without  the  necessary  programming/analysis  skills  would  fall 
into  the  levels  1  or  2.  Those  personnel  who  have  much  experience 
with  the  subject  software  system,  and  generally  whose  capabilities 
are  considered  critical  tp  the  support  of  the  subject  software  system 
fall  into  the  levels  4  or  5. 

b.  The  %  DEDICATED  is  the  percentage  of  time  which  the  support 
personnel  are  dedicated  to  the  subject  software  system  release  as 
opposed  to  some  other  releases  or  other  software  systems.  If  the 
personnel  were  working  only  on  the  subject  software  release,  then 
this  percentage  would  be  100. 

Example:  As  an  extended  example,  if  nine  F-16  SMS  organic 

support  personnel  were  distributed  so  that  5  (level  2)  were  100% 
dedicated,  3  (level  3)  were  75%  dedicated,  and  1  (level  4)  was  50% 
dedicated,  then  the  data  would  be  listed  as  follows: 


Organic  %  Dedicated  Contractor  %Dedicated 


Skill  level 
Skill  level 
Skill  level 
Skill  level 
Skill  level 
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3. 6. 1.8  RELEASE  ID/VERSION.  The  RELEASE  ID/VERSION  is  a  unique 
identifier  for  the  subject  software  release  being  reported. 

Example:  SF1 

3.6. 1.9  RELEASE  START  DATE.  The  RELEASE  START  DATE  is  that  date 
when  major  analysis  activity  related  to  the  subject  software  release 
begins  for  which  software  support  personnel  are  required. 

Example:  01/01/83 

3.6.1.10  RELEASE  ENGINEERING  COMPLETION  DATE.  The  RELEASE 
ENGINEERING  CO^LETION  OATE  is  that  date  when  the  software 
engineering  part  of  the  release  is  complete.  The  activities 
completed  for  the  project  block  release  include  design,  code,  unit 
test,  integration  test,  and  operational  test  and  evaluation.  Time 
for  "kit"  proofing,  prom  burning,  modification  of  technical  orders, 
and  other  such  activities  which  typically  occur  between  the 
engineering  completion  and  actual  field  implementation  is  not 
included.  A  typical  release  cycle  migirt  be  18  months  with  the  first 
12  months  for  software  engineering  and  the  last  6  months  for  tech¬ 
nical  order  preparation  and  field  distribution. 

Example:  06/01/83 

3.6.1.11  RELEASE  FIELD  DATE.  The  RELEASE  FIELD  DATE  is  that  date 
when  the  subject  software  release  is  officially  distributed  to  the 
field  for  operational  use.  See  paragraph  3.6.1.10  for  some  further 
discussion.  If  the  release  was  never  fielded,  specify  NOT  FIELDED. 

Example:  09/01/83 


3.6.1.12  CHANGE  DATA.  The  CHANGE  DATA  is  classification  and  effort 
information  on  each  software  change  request  (correction  of  defi¬ 
ciency,  enhancement  due  to  addition  or  deletion  of  a  capability,  or 
conversion  change  due  to  an  external  system  requirement  modification) 


in  the  subject  software  release.  Individual  fields  are 
below,  and  an  example  is  shown  in  figure  3-3. 


EXAMPLE  OF  CHANGE  OATA 


II I -23 


4 

•>'/ */ VwV.v./ 'A-“,y1./. -\V 


THE  BDM  CORPORATION 


3DM/A-35-G510-TR 


» 


(1)  Change  Request  ID.  Unique  identifier  for  the  change 
request. 

(2)  Configuration  Management  Open  Date.  Date  change  request 
was  opened  by  configuration  management. 

(3)  Configuration  Management  Close  Date.  Date  change  request 
was  closed  by  configuration  management. 

(4)  Actual  Person  Months.  Actual  person  months  to  complete 
the  change  request 

(5)  Change  Request  Type.  Correction  (C),  Enhancement  (E), 
Conversion  (V). 

(6)  Change  Request  Complexity.  Low  (L),  Medium  (M),  High  (H) 


(7)  Change  Request  Priority. 
Emergency  (E). 


Normal  (N),  Urgent  (U), 


See  the  glossary  for  further  definition  of  the  change  request  type, 
complexity,  and  priority. 

3.6.2  Data  Collection  Procedure. 

a.  It  is  recommended  that  all  Air  Force  software  support  sites 
(AFLCs,  MAJCOM  support  facilities)  collect  the  minimal  data  items 
described  in  paragraph  3.6.1  for  each  official  software  release  of 
each  mission  critical  software  system  being  supported.  Data 
collection  would  begin  at  any  point  in  the  software  life  cycle  when  a 
sustained  software  support  effort  with  uniquely  identified  software 
releases  was  being  accomplished  under  direct  control  of  the  site 
management.  The  actual  support  personnel  could  be  organic, 

contractor  or  some  combination.  Typically  this  would  occur  at  some 


1 1 1-24 


THE  BOM  CORPORATION 


BDM/A-85-0510-TR 


■ 

Niy 


S’ 


time  after  Initial  Operational  Capability  and  no  later  than  Program 
Management  Responsibility  Transfer. 


b.  The  specific  data  collection  form  could  be  something  like  that 
illustrated  in  figure  3-2  as  a  temporary  measure,  but  would  probably 
need  to  evolve  into  an  official  Air  Force  form.  There  might  be  a 
requirement  to  integrate  this  information  (format  and  content)  with 
the  requirements  of  related  Air  Force  regulations  and  practices  such 
as  AFM  66-1,  Maintenance  Management  (Reference  8.13),  or  AFR  800-18, 
Air  Force  Reliability  and  Maintainability  Program  (reference  8.14). 


c.  Of  prime  importance  is  the  integration  of  the  collected  data 
with  the  current  data  described  in  this  report.  AFOTEC  personnel 
need  to  have  a  straightforward  approach  to  obtaining  and  merging  new 
release  data  into  the  release  data  base  (see  section  on  BASE  III  data 
bases)  and  updating  the  maintenance  profiles  based  upon  the  updated 
data.  In  addition,  any  statistical  analysis  based  upon  changed  data 
items  such  as  predicted  person  months  per  change  would  need  to  be 
updated.  As  a  part  of  the  data  collection  procedure,  it  is 
recommended  that  AFOTEC  develop  the  appropriate  procedures  and  tools 
to  facilitate  this  regular  update  process. 


d.  A  top  level  view  of  the  data  collection  procedure  and  AFOTEC 
update  of  the  maintenance  profiles  is  illustrated  in  figure  3-4. 
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SECTION  IV 
RESULTS  OF  ANALYSIS 

4.1  INTRODUCTION. 

a.  This  section  reports  on  a  variety  of  analyses  conducted  on  the 
software  supportabi 1 ity  and  maintenance  data  collected  during  this 
effort.  With  the  exception  of  histograms  of  available  person-time, 
which  are  presented  as  Maintenance  Profiles  in  Section  V,  all  analy¬ 
tical  results  are  presented  in  this  section. 

b.  The  goals  of  the  analysis  conducted  during  this  study  include: 

(1)  Verify  that  consistent  data  on  system  background,  evalua¬ 
tion  support  and  risk  metrics,  and  system  maintenance 
activity  has  been  collected 

(2)  Reduce  data  to  usable  form,  consistent  across  systems 

(3)  Enter  data  into  a  dBASE  III  format  on  an  IBM-compatible 

microcomputer  for  ease  of  analysis,  reporting,  and  entry 
to  the  3MDP  statistical  analysis  package.  (The  BMDP 

package  used  was  resident  on  a  VAX  11/780  computer.) 

(4)  Determine  accuracy  of  data  and  general  confidence  in  data 

(5)  Produce  summary  reports  of  raw  and  computed  data 

(6)  Build  profiles  of  software  maintenance  activity  data 

(7)  Conduct  statistical  analyses  of  data 


(8)  Derive  conclusions  concerning  impact  upon  the  current 
Risk  Assessment  Methodology  for  Software  Supportabi 1 ity. 
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c.  A  summary  of  the  status  of  these  goals  is  presented  in 

table  4-1. 

4.2  OVERVIEW  OF  ANALYSIS  ENVIRONMENT. 

This  section  contains  a  brief  description  of  the  environment 
in  which  the  data  collected  during  this  study  effort  was  organized, 
analyzed,  and  reported. 

4.2.1  Hardware/Sof tware  System.  The  hardware  and  software  used  to 

support  the  data  analysis  are  illustrated  in  figure  4-1.  The  primary 
data  storage,  organization  and  summary  analysis  were  done  using  the 
d8ASE  III  data  base  management  software  (reference  8.15)  on  an  IBM  PC 
with  two  floppy  disks  and  a  dot  matrix  printer.  A  SmarTerm  220  com¬ 
munication  terminal  emulator  (reference  8.16)  was  used  to  provide  the 
capability  to  transfer  data  files  from  the  IBM  PC  to  a  VAX  11/780.  - 
The  bulk  of  the  statistical  analyses  was  conducted  on  the  VAX  using 
the  BMDP  statistical  software  package  (reference  8.17). 

4.2.2  dBASE  III  Analysis  Data  Files. 

a.  All  of  the  data  are  organized  onto  three  disks  using  dBASE  III 
on  an  IBM  PC.  Disk  1  mainly  contains  the  data  taken  off  the  evaluation 
form  (see  appendix  C).  Disk  2  contains  the  maintenance  data  col¬ 
lected  on  software  block  releases.  Disk  3  contains  system  identifi¬ 
cation  information. 

b.  dBASE  III  is  a  relational  model  for  data  base  management 
systems.  It  is  used  here  to  (1)  structure  data  bases,  (2)  facilitate 
data  entry  and  management,  (3)  produce  reports,  and  (4)  obtain  pre¬ 
liminary  analysis  results. 
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Table  4-1. 

Summary  of  Analysis  Results  and  Status 

CONFIDENCE 


GOAL 

STATUS 

'DATA/PR0CESS1 

Verify 

Oata 

Complete . 

Low/Medium 

Reduce 

Oata 

Complete. 

Medium/Medi urn 

Build 

Data  Base 

Complete. 

Medi um/High 

Analyze 

Data 

Validity 

Complete. 

Low/Med i urn 

Report 

Data 

Complete.  Data  reports  (appendix  D)  and 
analysis  reports  (section  IV)  have  been 
generated. 

Medi.um/High 

Build 

Prof i les 

Basic  required  profiles  (all  systems,  by 
site,  by  software  type)  have  been  generated. 

Medium/High 

Analyze 

Data 

Analysis  on  profiles,  correlation  of  support 
metric  to  risk,  and  correlation  of  categories 
to  person  time  is  complete.  Factor  and  regression 
analyses  are  complete. 

Medium/High 

Determine 

Methodology 

Impact 

BASELINE  SUPPORT  PROFILE 

1.  Basic  profile  data  (type,  complexity, 
priority)  has  not  changed.  Availability 
of  conversion  data  was  limited  by  data 
recording  technique,  not  by  actual 
functional  use.  Priority  data  did  not  seem 
to  be  universally  important  in  the  mission 
critical  categories  of  emergency,  urgent, 
normal.  Prioritizing  the  change  requests 
did  seem  to  be  universal,  nowever. 

Medium/Medium 

2.  The  historical  profile  has  changed  substan¬ 
tially  in  form.  Focus  is  upon  individual 
block  release  and  the  available  person  time 
per  change.  As  more  accurate  data  is 
available,  actual/estimated  person  time 
and  personnel  skill  levels  can  be  used. 

As  more  accurate  data  is  available,  change 
type,  complexity,  and  priority  can  be  useful. 
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GOAL 


Table  4-1. 

Summary  of  Analysis  Results  and  Status  (Concluded) 


CONFIDENCE 

STATUS  (DATA/PROCESS) 

3.  Data  is  not  easily  available,  but  could  be. 

4.  Data  is  not  consistent,  but  could  be  much 
better. 

5.  Data  collected  needs  to  be  centrally  located 
in  a  data  base  for  historical  baseline. 

6.  Derivation  of  a  baseline  support  profile  will 
not  be  difficult;  agreement  on  such  a  baseline 
by  user/supporter  will  be  most  difficult. 

EVALUATION  PROCEDURE  Medium/Medium 

1.  Calibration  will  be  very  important. 

2.  Product  evaluation  will  not  be  significantly 
different. 

3.  Environment  evaluation  will  be  impacted  by 
integrating  the  baseline  support  agreement. 

4.  Life  cycle  evaluation  has  been  mentioned 
by  several  evaluators  as  very  important. 

There  seemed  to  be  more  interest  in  this  part 
than  expected,  and  very  little  resistance 

or  difficulty  in  completing  the  survey  form. 

AFOTEC  needs  to  implement  this  evaluation. 

5.  Procedure  for  continuing  maintenance  data 
collection  has  been  recommended. 

EVALUATION  METRIC  TO  RISK  CONVERSION  Medium/High 

1.  Linear  Conversion  Model.  Analysis  of  data 
indicates  a  poor  fit. 

2.  Logistic  Transform  Model.  Analysis  of  data 
indicates  a  better  fit  than  the  linear  model. 

3.  Factor  Regression  Model.  Analysis  of  data 
indicates  which  factors  are  drivers  for 
determining  risk. 
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Figure  4-1.  Hardware/Software  Analysis  Support  System 
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c.  The  data  files  are  transmitted  via  modem  using  a  SmarTerm  22 0 
disk  in  the  IBM  PC  to  a  VAX  for  analysis  using  the  BMDP  statis¬ 
tical  software. 

4.2.2. 1  Disk  I:  Software  Supoortabi 1 ity  Evaluation  Data.  This  disk 
contains  56  files.  A  one  line  general  description  for  each  file  is 
contained  in  the  file  F I LEDESC .DBF  indexed  by  the  file  FILEDESC.NCX. 
The  four  primary  data  bases  are:  SYSINFO,  EVALG,  EVALD,  and  EVALC. 
They  are  described  in  the  following  four  sections. 

4.2.2. 1. 1  SYSINFO:  Software  System  Information.  This  data  base  i s 
an  identical  copy  from  Disk  3.  See  the  Disk  3  description  for 
further  information. 

4.2.2. 1.2  EVALG:  General  Descriptive  Information.  This  data  base 
has  29  items  of  information  primarily  concerned  with  the  background 
data  collected  from  part  1  .'2  of  the  site  survey  form  (see 
appendix  C) .  EVALG  is  ordered  by  an  evaluation  identification  number 
(EVALID)  using  the  index  file  EVALIDG.NDX.  EVALG  is  linked  to  tne 
SYSINFO  data  base  by  software  system  identification  number  (SWSVSID' 
to  access  software  system  information. 

4.2.2. 1.3  EVALD:  Evaluation  Data  On  Systems  At  Delivery  Time.  This 
data  base  has  48  items  of  evaluation  data  on  systems  "at  delivery" 
from  part  2  of  the  site  survey  form  (see  appendix  C).  EVALD  is  also 
ordered  by  evaluation  identification  number  (EVALID),  using  the  index 
file  EVALIDD.NOX.  EVALD  is  linked  to  the  SYSINFO  data  base  by  soft¬ 
ware  system  identification  number  (SWSYSID)  to  access  software  system 
information.  It  can  also  link  to  the  EVALG  data  base  by  EVALID  to 
access  general  evaluation  information. 

4.2.2. 1.4  EVALC:  Evaluation  Data  on  Systems  at  Current  Time.  This 
data  base,  identical  in  structure  to  EVALD,  has  48  items  of  evalua¬ 
tion  data  on  "current"  systems  from  part  2  of  the  site  survey  form 
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(see  appendix  C).  EVALC  is  ordered  by  evaluation  identification 
number  (EVALID)  using  the  index  file  EVALIDC.NDX.  It  is  linked  to 
the  SYSINFO  data  base  by  software  system  identification  number 
(SWSYSID)  to  access  software  system  information.  It  also  can  link  to 
the  EVALG  data  base  by  EVALID  to  access  general  evaluation 


information. 


4. 2. 2. 2  Disk  2:  Software  Maintenance  Block  Release  Summary  Data. 
This  disk  contains  21  files.  A  one  line  general  description  for  each 
file  is  contained  in  the  file  F I LEDESC . DBF  indexed  by  the  file 
FILEDESC.NDX.  The  two  primary  data  bases  are  SYSINFO  and  RLS_SMRY. 
They  are  described  in  the  following  two  sections. 


4. 2. 2. 2.1  SYSINFO:  Software  System  Information.  This  data  base  is 
an  identical  copy  from  Disk  3.  See  the  Disk  3  description  for 
further  information. 


4. 2. 2. 2. 2  RLSwSMRV:  Software  System  Block  Release  Summary  jata. 
This  data  base  has  21  items  of  information  on  software  maintenance 
block  releases,  corresponding  to  data  described  in  Dart  3  of  the  site 
survey  form  (see  appendix  C).  RLS_SMRY  is  ordered  by  software  system 
identification  number  (SWSYSIO)  and  release  identir ication  number 
(RLSID)  using  index  file  RLS_SMRY .NDX.  It  is  linked  to  the  SYSINFO 
data  base  by  SWSYSID  to  access  software  system  information. 


4.2.2. 3  Disk  3:  System  Identification  Information.  This  disk  con¬ 
tains  19  files.  A. one  line  general  description  for  each  file  is  con¬ 
tained  in  the  file  FILEDESC.OBF  indexed  by  the  file  FILEDESC.NOX. 
The  two  primary  data  bases  are:  SYSINFO  and  SWSYSDES.  They  are 
described  in  the  following  two  sections. 


4.2.2 . 3. 1  SYSINFO:  Software  System  Information.  This  data  base 
contains  10  items  of  software  system  identif ication  data  collected  in 
part  1.1  of  the  site  survey  form  (see  appendix  C).  SYSINFO  ^s 
ordered  by  a  software  system  identification  number  (SWSYSID)  using 
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index  file  SWSYSID.NDX.  The  data  items  KLINES  and  PCHILEV  are  the 
system  averages  for  variables  of  the  same  name  in  the  general 
evaluation  data  base  (EVALG)  on  disk  1.  These  items,  along  with 
NSITE,  NSWTYPE,  and  AVGSKILL,  are  included  in  this  data  base  because 
they  are  needed  on  a  system  basis  for  analysis.  The  SYSINFO  data 
bases  on  both  Disks  1  and  2  are  identical  to  this  data  base. 

4.2.2. 3.2  SWSYSDES:  Software  System  Description.  This  data  base 
holds  a  brief  description  of  each  software  system  for  which  data  was 
collected  during  this  study  effort.  SWSYSDES  is  linked  to  the 
SYSINFO  data  base  by  software  system  identification  number  (SWSYSID) 
to  access  software  system  information.  It  is  ordered  by  software 
system  identification  number  (SWSYSID)  using  the  index  file 
SWSYSDES. NDX. 

4.2.3  BMDP  Statistical  Analysis  Software. 

a.  The  BMDP  computer  programs  are  designed  to  aid  data  analysis 
by  providing  methods  ranging  from  simple  data  display  and  description 
to  advanced  statistical  techniques.  Data  are  usually  analyzed  by  an 
iterative  "examine  and  modify"  series  of  steps.  First  the  data  are 
examined  for  unreasonable  values,  graphically  and  numerically.  If 
unreasonable  values  are  found  they  are  checked  and,  if  possible,  cor¬ 
rected.  An  analysis  is  then  performed.  This  analysis  may  identify 
other  inconsistent  observations  or  indicate  that  further  analyses  are 
needed.  The  BMDP  programs  are  designed  to  handle  all  steps  in  an 
analysis,  from  the  simple  to  the  sophisticated. 

b.  The  BMDP  programs  are  organized  so  the  problem  to  be  analyzed, 
the  variables  to  be  used  in  the  analysis,  and  the  layout  of  the  data 
are  specified  in  a  uniform  manner  for  all  programs.  This  permits 
different  analyses,  of  the  same  data  with  only  minor  changes  in  the 
instructions. 
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c.  See  reference  8.17  for  further  information  on  the  8MDP  soft¬ 
ware. 


4.3  BACKGROUND  DATA  ANALYSIS. 

At  every  site  visited,  survey  forms  like  the  one  shown  in 
Appendix  C  were  distributed  to  knowledgeable  software  maintenance 
personnel  for  completion.  For  virtually  all  of  the  software  systems 
encountered,  at  least  one  survey  form  was  completed.  Reported  in 
this  section  are  analysis  results  for  the  background  data  collected 
in  section  1  of  the  survey  forms  (see  page  C-6). 

4.3.1  Summary  Results.  Listed  below  are  summary  results  for  the 
major  areas  covered  under  background  data  for  all  the  systems  sur¬ 
veyed.  A  complete  (except  for  individual  evaluator  names)  list  of 
.background  data  is  presented  in  appendix  D.  A  summary  of  the  back¬ 
ground  data  results  by  the  major  areas  is  presented  in  table  4-2. 


4. 3. 1.1  Software  System  Types.  There  were  7  types  of  software 
systems  for  which  data  was  collected:  OFP,  C-E,  EW,  ATD,  ATE,  SIM, 
SUP.  Only  the  OFP,  C-E,  EW,  and  SUP  types  had  substantial  amounts 
of  maintenance  data.  Analysis  results  stratified  by  software  system 
type  are  presented  in  other  parts  of  section  4  and  in  section  5. 

4. 3. 1.2  Software  System  Size.  The  size  of  the  software  systems  in 
thousands  (K)  of  source  lines  being  supported  ranged  from  IK  to 
2.800K.  The  average  size'  (total  source  lines  divided  by  number  of 
software  systems  reporting  data)  is  181K.  The  average  number  of 
source  lines  supported  by  one  person  is  23K,  with  a  range  of  0.3K  to 
200K. 


4. 3. 1.3  Language  Usage.  The  predominant  source  language  for  soft¬ 
ware  systems  in  the  survey  is  assembly.  A  wide  variety  of  assembly 
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and  high  order  languages  (FORTRAN,  JOVIAL,  COBOL,  PL/I,  data  base, 
etc.)  were  in  use.  It  appears  that  the  more  recent  systems  (e.g., 
F-16)  have  a  higher  percentage  of  high  order  language  source. 

4. 3. 1.4  Development.  The  development  background  data  included 
development  period,  development  contractor(s) ,  and  approximate  level 
of  effort.  This  data  was  not  generally  known  except  for  the  develop¬ 
ment  contractor.  Five  or  six  contractors  tended  to  be  the  predomi¬ 
nant  developers. 

4. 3. 1.5  Life  Cycle  Events.  Life  cycle  events  which  had  major  impact 
upon  the  software  support  varied  from  political  concerns  to  tech¬ 
nology  concerns.  One  event  that  consistently  had  a  major  impact  was 
a  major  conversion  effort  due  to  application  hardware  modifications. 
Attrition  of  key  personnel  and  changeover  in  support  contractors 
were  other  major  events.  The  contractor  change  impact  was  bad  or 
good  depending  upon  the  circumstances.  •  Official  government  PMRT  was 
also  considered  to  be  a  major  event,  usually  with  positive  impact. 

4.3. 1.6  Personnel .  The  number  of  personnel  supporting  a  software 
system  varied  from  one  to  84.  The  average  number  of  source  lines  (k) 
supported  by  one  person  was  23K  with  a  range  from  0.3K  to  200K. 

4. 3. 1.7  Support  Systems.  Background  data  on  support  systems 
included  a  list  of  (or  reference  to  a  list  of)  support  hardware/sof t- 
ware  and  an  indication  of  the  percentage  of  time  the  support  systems 
were  available  for  use.  Although  the  data  was  pretty  limited,  it  was 
apparent  that  a  wide  variety  of  hardware  and  software  support  was 
being  utilized  with  a  part  of  the  support  systems  (especially 
integrated  laboratory  equipment)  application  specific.  Frequently 
support  systems  were  being  used  by  several  groups  including  users  and 
training  personnel.  Quite  often  availability  of  support  systems  was 
considered  a  problem. 
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4. 3. 1.8  Problems.  Problems  most  frequently  mentioned  were  lack  of 
qualified  personnel,  inability  to  keep  qualified  personnel,  quality 
of  documentation,  overhead  management  and  external  time  demands,  con¬ 
figuration  management,  and  inter-organization  interfaces. 

4.4  EVALUATION  DATA  ANALYSIS. 

a.  The  evaluation  data  consists  of  a  high  level,  subjective 
rating  of  each  software  system's  support  aspects  and  an  estimate  of 
the  software  system's  overall  supportabi 1 ity  risk  (see  appendices  C 
and  0).  Evaluation  data  were  obtained  relative  to  two  different 
points  in  time:  delivery  of  the  software  for  support,  and  current 
time.  In  the  case  where  delivery  was  essentially  the  same  as  current 
time,  the  two  evaluations  were  the  same.  One  evaluation  per  system 
was  completed  by  a  lead  support  person  for  the  system,  (for  a  few 
systems,  several  evaluations  were  done  by  several  evaluators). 

b.  There  were  several  problems  with  the  evaluation  data  which 
contribute  to  the  generally  low  confidence  in  the  absolute  accuracy 
of  this  data.  These  problems  are  summarized  below: 

(1)  Missing  Data.  Some  of  the  evaluators  for  the  systems 

surveyed  early  in  the  study  left  blanks.  This  missing 

data  is  very  difficult  to  integrate. 

(2)  Focus  of  Risk.  The  early  evaluations  did  not  focus  on 

the  specific  concept  that  the  risk  is  based  upon  comple¬ 
tion  of  a  block  release  within  the  resources  (e.g.,  time, 
systems,  and  personnel)  assigned  at  the  start  of  the 
block  change  cycle.  Most  of  this  problem  was  due  to  the 
evolution  of  the  survey  process  itself.  Generally  this 
means  that  even  if  the  supportabi 1 i ty  metrics  indicated 
good  scores,  the  risk  was  contradictori ly  estimated  to  be 
high  (e.g.,  1.0).  In  later  surveys,  the  evaluators  were 
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more  carefully  instructed  as  to  the  meaning  of  the  risk 
and  consequently  the  estimations  of  risk  were  more 
closely  related  to  the  supportability  metrics. 

(3)  Terminology  Consistency.  As  with  all  subjective  evalua¬ 
tions  it  is  necessary  to  carefully  explain  terminology 
and  provide  for  calibration,  review,  and  frequent  delphi- 
like  corrections.  These  evaluations  had  very  little 
benefit  of  these  techniques.  The  anticipated  review  of 
the  data  by  the  evaluators  in  the  context  of  this  report 
and  in  comparison  to  other  systems  should  provide  some 
improvement  in  the  evaluation  data  consistency,  accuracy, 
and  confidence  in  the  results. 

c.  A  general  observation  based  upon  the  evaluation  data  and  the 
survey  process  is  that  an  evaluation  can  be  conducted  relative  to  a 
support  baseline.  However,  a  good  explanation  of  the  terminology, 
process,  and  desired  focus  is  essential  during  the  calibration  phase. 
Whenever  this  was  done  during  the  survey  effort,  the  evaluation  data 
seemed  to  be  more  consistent  and  within  the  bounds  of  what  the 
methodology  would  predict  (e.g.,  high  support  metrics  associated  with 
a  low  risk  estimate).  It  appears  that  the  software  product 
evaluations  will  not  be  much  affected  by  the  nature  of  the  support 
baseline,  while  the  software  support  facility  evaluation  and  the 
software  life  cycle  management  evaluation  will  be  more  directly 
affected.  It  is  not  understood  how  sensitive  each  of  the  evaluations 
would  be  to  changes  in  the  baseline  profile.  It  is  doubtful  that  any 
of  the  data  collected  in  the  survey  would  allow  for  any  such 
conclusions.  This  sensitivity  would  be  a  good  analysis  issue  to 
resolve  during  a  pilot  study  where  the  methodology,  complete  with 
evaluations,  is  applied  in  a  more  consistent  and  controlled  manne’-. 

d.  Evaluation  ratings  of  45  aspects  of  software  system  support- 
ability  were  requested  from  survey  respondents  (evaluators)  in 
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section  2  of  the  survey  form  (see  pages  C-7  and  C-8).  Evaluation 
ratings  were  collected  on  the  software  systems  as  they  were  at 
delivery  and  as  they  were  in  their  current  forms  at  the  times  the 
survey  forms  were  administered.  Since  the  current  system  evaluation 
data  is  of  more  immediate  interest,  and  since  the  "at  delivery" 
evaluation  data  is  generally  less  reliable  (because  ratings  were 
generally  not  collected  at  or  soon  after  delivery,  but  were  based  on 
corporate  memory  of  the  system  status  at  delivery  some  time  in  the 
past),  the  analyses  of  this  section  treat  exclusively  the  ratings  of 
current  systems.  Because  these  analyses  examine  the  internal  rela¬ 
tionships  among  the  supportabi lity  aspects  rated,  and  the  relation¬ 
ships  should  not  differ  essentially  between  delivered  and  current 
systems,  it  is  anticipated  that  similar  results  would  be  obtained  if 
these  analyses  were  to  be  conducted  on  the  ratings  of  systems  at 
delivery. 

e.  Of  the  45  evaluation  ratings  provided  for  on  the  survey  form, 
the  first  44  attempt  to  measure  aspects  of  system  supportabi 1 ity  that 
are  supposed  to  affect  the  risk  of  not  being  able  to  support  a  given 
system.  The  last  evaluation  rating  is  a  direct  rating  of  that  risk 
as  it  is  perceived  by  the  evaluator.  It  is  the  goal  of  this  section 
to  explore  and  compare  methods  for  converting  supportabi 1 ity  ratings 
into  a  risk  measure. 

4.4.1  Risk  Versus  General  Software  Supportabi  1  ity  Rating.  The 
general  software  supportabi 1 i ty  rating  (labelled  as  2.4.1  in  the 
survey  form)  was  intended  to  measure  the  general  supportabi 1 i ty  of 
the  software  system.  In  this  context,  it  was  expected  that  the 
evaluator  would  in  some  fashion  mentally  integrate  the  ratings 
assigned  to  all  the  lower  level  aspects  of  supportabi 1 i ty  to  arrive 
at  a  general  supportabi 1 i ty  rating.  Of  course,  it  is  also  possible 
that  the  evaluator  will  integrate  into  the  general  supportaoi 1 i ty 
rating  aspects  of  supportabi 1 ity  which  are  not  captured  in  the  list 
of  lower  level  aspects  covered  by  the  survey  form.  Ideally,  the 
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evaluator's  risk  rating  should  closely,  in  an  inverse  manner,  reflect 
the  general  supportabi 1 i ty  rating;  that  is,  if  general  supportabi 1 ity 
is  rated  high,  risk  should  be  rated  low,  and  vice  versa.  This  sec¬ 
tion  examines  the  relationship  between  general  supportabi 1 ity  rating 
and  risk  rating  and  means  of  converting  general  supportabi 1 ity  rating 
to  risk. 

4.4. 1.1  A  Simple  Linear  Conversion  Function.  One  way  to  convert 
general  supportabi 1 ity  rating  to  risk,  though  simple,  is  immediately 
obvious.  It  is  via  the  line  r  function 

R  ■  1  -  (G  +  50)  /  100, 

where 

R  =  risk,  and 

G  =  general  supportabi 1 ity  rating. 

This  method  is  illustrated  in  figure  4-2  where  actual  data  points 
collected  from  the  site  survey  form  are  plotted.  The  horizontal  axis 
of  figure  4-2  is  general  supportabi 1 ity  rating  (labelled  ASUPPORT), 
and  the  vertical  axis  is  risk  (labelled  ARISK).  The  numerals  in  the 
plot  indicate  the  number  of  data  points  occurring  at  a  location 

(there  is  a  total  of  88  data  points  available  from  the  set  of  survey 
forms).  The  diagonal  straight  line  in  the  plot  represents  the  linear 
function  given  above.  It  is  evident  from  the  plot  that  this  straight 
line  does  not  reflect  the  curved  nature  of  the  data,  so  that  while 

the  line  does  seem  to  fit  the  data  at  the  upper  left  and  lower  right 

corners  of  the  plot,  it  clearly  does  not  fit  well  the  data  between 
the  two  corners.  An  alternative  conversion  function  is  needed. 

4. 4. 1.2  A  Linear  Regression  Approach. 

a.  There  are  numerous  ways  to  arrive  at  a  curvilinear  function 

representing  the  relationship  between  risk,  R,  and  general  support- 
ability  rating,  G.  One  way  would  be  to  fit  directly  a  curvilinear 
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function,  R  =  f(G),  involving  a  polynomial,  say.  A  difficulty  of 
this  approach  is  that  it  is  not  clear  what  form  f(G)  should  take,  and 
interpretation  may  not  be  easy  if  f(G)  becomes  very  complex.  Another 
approach  is  to  fit  a  function  h(G)  to  some  transformation,  T,  of  R, 
so  that  the  relationship  becomes  T(R)  =  h(G).  In  certain  cases,  the 
function  T(R)  *  H(G)  may  be  much  simpler  and  more  readily  interpre¬ 
table  than  the  function  R  =  f(G).  The  approach  employing  a  trans¬ 
formation  of  risk  is  used  in  the  regression  analysis  below. 

b.  Note  that  risk  values  may  not  fall  below  zero,  nor  exceed  one. 
These  constraints  should  also  be  obeyed  by  any  functional  forms  used 
to  represent  or  predict  risk.  For  example,  a  risk  of  1.1  predicted 
by  a  function  for  a  general  supportabi 1 ity  rating  of  -45  would  be 
inadmissible.  A  transformation  of  risk  that  ensures  these  con¬ 
straints  will  be  met  is  the  logistic  transformation, 

T(R)  -  In  (R  /  (1  -  R)). 

With  R  taking  values  between  zero  and  one,  T(R)  will  take  values 
between  minus  infinity  and  plus  infinity.  To  see  that  the  transfor¬ 
mation  meets  the  constraints,  note  that  in  the  inverse  transformation 

T*1  ( T( R) )  =  R  =  1  /  (1  +  exp  (-T(R)), 

when  T(R)  tends  toward  minus  infinity,  R  will  tend  to  zero.  As  T(R) 
goes  to  plus  infinity,  R  will  go  to  one.  For  T(R)  equal  to  zero,  R 
will  equal  one-half. 

c.  Since  T(R)  is  undefined  for  R  equal  to  zero  or  one,  and  since 
R  may,  and  in  fact  does,  assume  those  va  ues,  it  is  necessary  to 
modify  T(R)  so  that  it  is  defined  for  R  equal  to  zero  or  one.  Here, 
the  values  of  R  are  simply  scaled  inward  slightly  from  the  interval 
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endpoints  zero  and  one  before  T(R)  is  applied.  The  scaling  function 
used  is 

R*  *  R  (1  -  a)  +  (a  /  2), 

where  a  is  a  small  positive  number.  R‘  thus  scales  values  between  0 
and  1  to  values  between  (a  /  2)  and  1  -  (a  /  2),  and  T(R')  is  defined 
for  all  values  of  R,  including  0  and  1.  The  overall  transformation 
of  R,  then,  is 


L<R>  =T‘R‘>  - ln  i  trTTrri  t’A-lY, 


For  this  analysis,  a  value  of  a  *  .001  was  arbitrarily  chosen. 

d.  The  data  of  figure  4-2  are  portrayed  in  figure  4-3  with  trans¬ 
formed  risk  values  L(R)  on  the  vertical  axis  (labelled  LRISK) .  From 
the  scatter  of  the  data,  it  appears  quite  reasonable  to  use  a 
straight  line  to  represent  the  relationship  between  transformed  risk 
(L)  and  general  support abi 1 i ty  rating  (3).  The  straight  line  regres¬ 
sion  model  to  be  used  is 

L  -  bg  +  b^  G  +  e, 

where  e  is  a  random  error  variable  and  bg  and  b^  are  parameters  to  be 
estimated.  Fitting  the  aDove  model  via  least  squares,  the  solid 
straight  line  in  figure  4-3  is  obtained,  the  equation  for  which  is 

L  =  .65011  -  .06674  G. 


This  equation  accounts  for  R* 
of  L  values  about  their  mean. 


-  .,35,  or  35  percent,  of  the  variation 
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e.  To  provide  a  better  view  of  what  has  been  gained  by  this 
approach  over  that  of  previous  section,  both  models  are  depicted  on 
the  risk  scale  in  figure  4-4.  The  straight  line  is  the  linear  con¬ 
version  function  of  section  4. 4. 1.1,  and  the  curved  line  is  the  model 
of  this  section  detransformed  to  show  risk  values  rather  than  trans¬ 
formed  risk  values.  A  transformed  value  L  is  detransformed  to  risk 
by  the  function 

R  =  (d  +  exp  (-L))-1  -  (a  /  2)}  /  (1  -  a). 

The  curved  line  better  reflects  the  scatter  of  the  data  than  does  the 
straight  line.  Also  noteworthy  is  the  fact  that  neither  model,  when 
used  for  converting  general  supportabi 1 ity  rating  to  risk,  will  yield 
risk  values  less  than  zero  or  greater  than  one. 

f.  The  conversion  of  several  general  supportabi 1 ity  metrics  to 
risk  using  the  linear  model  and  the  linear  regression  transform  model 
is  illustrated  in  table  4-3. 

Table  4-3. 

Conversion  of  General  Supportabi 1 i ty  Metric  to  Risk 
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4.4.2  Consolidation  of  the  Supportability  Ratings  into  a  Few 
Supportabi 1 ity  Factors. 

a.  Section  4.4.1  examined  the  conversion  of  general  support- 
ability  rating  to  risk  via  two  different  functions.  While  either  of 
those  functions  is  relatively  easy  to  use  in  obtaining  a  risk 
estimate  from  a  general  supportabi 1 ity  rating,  both  rely  solely  on 
information  available  in  the  general  supportabi 1 ity  rating  and  ignore 
information  contained  in  the  other  43  supportabi 1 ity  ratings 
(laDelled  as  2.1.1  through  2.3.3  in  the  survey  form).  Although  the 
general  supportabil ity  rating  is  intended  to  rate  overall  support- 
ability,  it  relies  on  an  evaluator's  conscious  or  subconscious  scheme 
for  integrating  all  aspects  of  supportabi 1 ity  and  rating  them  with 
one  number.  Clearly,  the  importance  attached  to  individual  aspects 
of  supportabil ity  in  arriving  at  a  general  supportability  rating  will 
vary,  perhaps  considerably,  from  one  evaluator  to  another  in  an 
unknown  way.  It  is  therefore  desirable  to  develop  a  means  of  objec¬ 
tively  (as  opposed  to  subjectively)  integrating  the  ratings  of  all 
the  supportability  aspects  covered  by  the  survey  form.  The  sections 
below  describe  such  an  approach. 

b.  To  set  the  stage  for  the  next  two  sections,  some  initial 
results  are  presented  here.  Shown  in  figure  4-5  are  correlation 
coefficients  for  pairs  of  rating  variables  that  are  related  through 
the  hierarchical  structure  of  the  survey  form  in  appendix  C.  The 
variable  names  are  straightforward  abbreviations  of  the  rating  cate¬ 
gories  listed  in  the  survey  form.  At  the  left  side  of  figure  4-5  are 
the  lowest  level  categories,  at  the  right  is  the  highest  level  cate¬ 
gory.  In  the  variable  name  APDOCMOD,  for  example,  "P"  indicates  the 
second-level  category  "product",  "DOC"  the  third-level  category 
"documentation"  within  "product",  and  "MOD"  the  fourth-level  category 
"modularity"  within  "documentation"  within  "product".  The  variables 
in  level  4  are  listed  in  the  same  order  as  their  respective  cate¬ 
gories  in  the  survey  form;  thus,  APDOCMOD  reflects  the  category 
labelled  2. 1.1.1  in  the  survey  form. 
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c.  Note  the.  general  tendency  for  the  correlations  between  levels 
to  increase  in  going  from  the  highest  level  (1)  to  the  lowest 
level  (4).  This  pattern  reflects  the  organization  of  the  survey  form 
in  that  lower  levels  focus  on  more  closely  related  supportabi 1 ity 
aspects  than  do  higher  levels. 

d.  The  correlations  in  figure  4-5  were  computed  by  the  BMDP  pro¬ 
gram  BMDPAM,  using  a  "smoothing"  technique  that  compensates  for 
missing  values  in  the  data  set,  so  while  these  correlations  are  for 
the  most  part  close  to  correlations  obtained  for  the  subset  of  data 
analyzed  in  the  following  sections,  they  are  not  identical.  These 
smoothed  correlations  do  serve,  however,  to  illustrate  the  hierarchi¬ 
cal  structure  just  discussed.  In  the  following  analysis,  some  of  the 
variables  listed  here  will  be  omitted  due  to  a  large  number  of  mis¬ 
sing  values. 

4. 4. 2.1  Analysis  Approach. 

a.  The  technique  used  in  this  analysis  to  consolidate  the  many 
supportabi 1 ity  ratings  is  known  as  factor  analysis.  Because  there 
are  so  many  (44)  supportabi 1 i ty  rating  variables  deriving  from  the 
survey  form,  it  is  extremely  difficult  to  grasp  directly  from  the 
ratings  data  the  relationships  among  the  rating  variables.  One 
important  feature  of  factor  analysis  is  that  it  provides  a  systematic 
method  for  reducing  the  dimensionality  of  a  large  set  of  variables  by 
producing  a  small  set  of  factors  that  retain  a  large  portion  of  the 
information  content  of  the  original  variables.  Another  notable 
feature  of  factor  analysis  is  that  it  can  be  used  to  discover  linear 
relationships  among  the  variables  and  to  suggest  the  identity  of 
basic  underlying  variables  that  the  factors  represent. 
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b.  The  mathematical  model  for  factor  analysis  is 

X1  =  Ml  Y1  +  +  MmYm  +  el 

x2  =  X2i  Yi  +  ...  +  X2mYrn  +  e2 


Xp  "  ^pl  Y1  +  •'*  +  ^pmYm  +  ep 

where  X^,  Xp  are  the  p  original  variables,  Y^,  Ym  are  the  m 

(unobservable)  factors,  is  the  so-called  loading  of  the  i-th 

variable  on  the  j-th  factor  (actually,  is  the  correlation  of  the 
i-th  variable  with  the  j-th  factor),  and  e-  is  a  random  component. 

J 

For  a  complete  description  of  factor  analysis,  see  chapter  nine  of 
Morrison  (1976)  (reference  8.18);  chapter  8  is  also  relevant. 

c.  Computations  for  this  factor  analysis  were  accomplished  using 
the  BMDP  Statistical  Software  factor  analysis  program  BMDP4M.  See 
the  BMDP  manual  (1983)  (reference  8.17)  for  details  on  the  BMDP4M 
program. 

d.  Once  the  factor  loading  matrix  is  estimated,  orthogonal  trans¬ 
formations  of  the  loading  matrix  sometimes  yield  loading  vectors  that 
are  more  readily  interpretable  in  the  subject  context  but  retain 
their  ability  to  model  the  original  variables.  These  orthogonal 
transformations  result  in  a  rigid  rotation  or  reflection  of  the 
coordinate  axes  of  the  m-dimensional  factor  space,  hence  the  trans¬ 
formed  loadings  are  referred  to  as  "rotated  factor  loadings." 

e.  The  next  section  discloses  key  results  of  the  factor  analysis 
performed  on  the  evaluation  data. 
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a.  The  factor  analysis  of  the  evaluation  data  was  begun  by 
deciding  what  supportability  rating  variables  should  be  included. 
For  some  variables,  such  as  APD0C1NS,  so  many  of  the  total  of  97  data 
cases  (from  97  completed  evaluation  survey  forms)  had  missing  values 
(no  entry  made  by  the  evaluator)  that  deletion  of  those  cases  would 
have  left  too  few  cases  to  perform  the  analysis.  Since  estimating 
the  missing  data  seemed  inappropriate,  the  only  alternative  was  to 
omit  such  variables  from  the  analysis.  In  selecting  variables  to  be 
omitted,  the  goal  was  to  keep  as  many  variables  as  possible  in  the 
analysis  while  maintaining  a  fairly  large  set  of  cases.  The  result 
of  this  process  was  that  7  variables  were  eliminated:  APDOCINS, 
APSRCINS,  AEPERCON,  AESYS8EN,  AESYSLAB,  AESYSOPE,  AESYSOTH.  Listed 
in  figure  4-6  are  the  remaining^  37  variables.  Of  the  97  original 
data  cases,  70  were  complete  in  the  37  variables  and  were  used  in  the 
analysis, 

b.  Figure  4-6  shows  rotated  factor  loadings  for  6  factors.  Since 
the  number  of  factors  needed  to  appropriately  describe  this  data  set 
was  initially  unknown,  loading  matrices  for  models  ranging  from  4  to 
9  factors  were  examined  for  interpretability.  The  6-factor  model  was 
chosen  because  it  accounts  for  79  percent  of  the  total  variance  of 
the  37  rating  variables,  and  it  yields  factors  that  are,  overall, 
more  clearly  interpretable  than  those  of  any  of  the  other  models. 
Interpretations  assigned  to  the  six  supportabi 1 i ty  factors  are  given 
below  in  table  4-4. 

c.  The  interpretations  were  arrived  at  by  noting  for  each  factor 
which  variables  had  loadings  of  .5  or  greater  (these  are  flagged  by 
asterisks  in  figure  4-6)  and  characterizing  the  quality  that  those 
variables  collectively  seem  to  measure.  All  the  interpretations  are 
straightforward  except  for  that  of  factor  4,  which  is  less  so.  It  is 
interesting  to  note  that  rating  variables  that  are  related  by  virtue 
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RCTATEO  FACTOR  L0A0INGS 


FACTCR 

FACTCR 
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FACTCR 
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Figure  4-6.  Rotated  Factor  Loadings 
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of  the  hierarchy  in  the  survey  form  tend  to  load  heavily  on  only  one 
factor,  indicating  that  relationships  intended  in  the  survey  form  are 
reflected  in  the  data. 

Table  4-4. 

Interpretations  of  Supportabi 1 ity  Factors 


FACTOR  NUMBER 


INTERPRETATION 
Support  Management 


Product 


Personnel 
Organization 
Faci 1 ities 
Support  Systems 


d.  The  basic  accomplishment  of  this  factor  analysis  is  that  the 
dimensionality  of  the  supportabi 1 ity  rating  data  has  been  reduced 
from  37  variables  to  6  factors  having  reasonable  and  useful  inter¬ 
pretations.  With  most  of  the  information  content  of  the  original 
variables  consolidated  into  the  factors,  the  factors  can  be  used  in 
subsequent  analyses  in  place  of  the  variables,  as  will  be  shown  in 
the  next  section. 

e.  Although  some  subjective  choice  was  involved  in  the  develop¬ 
ment  of  the  factor  model,  the  model  is  applied  consistently  across 
all  data  cases,  and  it  therefore  represents  an  improvement  over  the 
general  supportabi lity  rating  above,  which--as  discussed  in 
section  4.4.2--is  based  on  a  different  unknown  model  for  each  case 
(i.e.,  evaluator).  The  factor  model  represents  a  further  improvement 
in  that  the  factors  incorporate  the  general  supportabi 1 i ty  rating 
variable,  ASUPPORT. 
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4.4.3  The  Relationship  Between  Risk  Rating  and  Supportabi  1  it1) 
Factors. 


In  section  4.4.2,  the  numerous  supportabi 1 ity  rating  variables 
were  consolidated  into  six  factors.  The  purpose  of  this  section  is 
to  examine  and  characterize  the  relationship  between  risk  rating  and 
the  supportabi 1 ity  factors  named  in  table  4-4.  Of  particular 
interest  is  the  question  of  which  factors  figure  into  an  evaluator's 
determination  of  risk,  and  to  what  degree. 

4. 4. 3.1  Analysis  Approach. 

a.  Results  presented  in  the  next  section  were  obtained  through 
regression  analysis.  The  mathematical  model  for  this  regression 
analysis  is 


Y  *  bo  +  blXl  +  •  •  •  +  b6X6  +  e’ 

where  Y  =  the  risk  rating  (transformed), 
Xj  =  the  score  for  the  i-th  factor, 
e  =  a  random  component, 


and  the  regression  coefficients  bQ,  .  .  .,  bg  are  parameters  to  be 
estimated.  (The  Xj  here  are  equivalent  to  the  Y.  in  the  factor 
analysis  model  of  section  4. 4. 2.1,  not  to  the  X.  of  that  model.)  The 
risk  rating  variable,  R,  is  transformed  via  the  transformation  l(R) 
described  in  section  4.4.1.2--for  reasons  discussed  there--to  obtain 
the  dependent  variable  Y  above. 

b.  An  alternative  approach  that  might  be  tried  would  be  to  use  as 
the  independent  variables  Xi  in  the  above  model  the  supportabi 1 i ty 
rating  variables,  instead  of  the  factors  derived  from  them.  Such  an 
approach,  however,  would  result  in  an  unwieldy  model  with  37  indepen¬ 
dent  variables.  Of  course,  a  small  subset  of  those  variables  could 
be  selected  for  inclusion  in  the  model,  but  the  variable  selection 
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itself  then  becomes  problematic  insofar  as  most  of  the  variables--and 
the  information  they  may  contain — would  be  discarded.  Use  of  factors 
as  independent  variables  avoids  this  dilemma  because  the  factors  each 
incorporate,  to  greater  or  lesser  degree,  information  from  an_  of  the 
rating  variables,  with  each  factor  (as  shown  by  the  factor  loadings 
in  figure  4-6)  emphasizing  a  different  subset  of  the  rating 
variables.  Furthermore,  the  factor-based  regression  model  is  far 
more  parsimonious  and  therefore  more  easily  interpreted. 

c.  The  same  70  data  cases  used  to  construct  the  factors  obtained 
in  section  4. 4. 2. 2  were  used  in  conducting  the  regression  analysis  of 
this  section.  For  each  case,  the  factor  scores  X •  were  computed  from 
the  rating  variables,  and  to  those  data,  along  with  the  transformed 
risk  ratings  Y,  the  above  regression  model  was  fitted.  Results 
appear  in  the  next  section. 

4. 4. 3. 2  Results. 


a.  Results  for  the  regression  analysis  of  transformed  risk  versus 
the  six  supportability  factors  are  shown  in  figure  4-7.  These 
results  were  generated  by  the  BMOP  program  3MDP1R:  Multiple  Linear 
Regression.  The  dependent  variable,  transformed  risk  (Y  =  L ( R ) ) ,  has 
the  variable  name  LRISK  in  the  figure.  In  the  analysis  of  variance 
table  near  the  middle  of  figure  4-7,  the  F  statistic  testing  the 
significance  of  the  regression  model  is  8.163  (under  the  column 
heading  "F  RATIO"),  a  value  significant  at  the  0.0001  level, 
indicating  that  the  regression  coefficient  for  at  least  one  of  the 
factors  is  significantly  different  from  zero.  Examination  of  the 
table  at  the  bottom  of  figure  4-7  shows  that  t  statistic  values  (in  the 
column  labeled  "T")  for  the  coefficients  of  four  of  the  factors  have 
significance  probabilities  (the  column  labeled  "P(2  TAIL)")  less  than 
.05.  The  three  factors  are  thus  significant  at  the  .05  level;  they 
are  flagged  by  asterisks  to  the  left  of  their  variable  names  in  the 
column  labeled  "VARIABLE".  The  variable  names  are  obvious  abbrevi¬ 
ations  of  the  interpretive  labels  given  to  the  factors  in  table  4-4. 
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b.  The  interpretation  given  to  these  results  is  that  the  four 
supportabi  1 ity  factors  called  Support  Management,  Product,  Personnel, 
and  Support  Systems  figure  significantly  in  the  evaluators' 
assessment  of  risk;  the  other  two  factors--Organization  and 
Faci lities— do  not  (at  least  for  the  evaluators  represented  in  this 
data  set).  Estimates  of  the  regression  coefficients  b^,  .  .  .,  bg, 
are  printed  in  the  second  column  (labeled  "COEFFICIENT")  of  the  table 
at  the  bottom  of  figure  4-7.  Note  first  that  the  coefficients  of  the 
significant  factors  are  all  negative.  This  means  that  higher  scores 
for  those  factors  are  associated  with  lower  transformed  risk  values-- 
and  correspondingly,  with  lower  risk  ratings.  Conversely,  lower 
factor  scores  are  associated  with  higher  risk  ratings.  The 
magnitudes  of  the  coeff icie^ts  indicate  the  relative  strengths  of  the 
corresponding  factors  in  this  association;  hence  "support  systems"  is 
in  this  sense  the  most  important  factor  in  rating  risk,  followed  by 
product,  support  management,  and  personnel. 

c.  Finally,  the  above  fitted  regression  model  has  an  R  statistic 

(labeled  "MULTIPLE  R-SQUARE"  in  figure  4-7)  of  .4374,  which  indicates 

that  the  model  accounts  for  about  44  percent  of  the  variation  of  the 

2 

transformed  risk  values  about  their  mean.  This  R  value  is  quite 
low.  An  alternative  view  is  that  the  model  fails  to  account  for  the 
other  56  percent  of  the  variation.  Evidently,  there  are  other 
aspects  not  assessed  in  the  survey  form  that  bear  on  risk  rating,  or 
there  is  a  set  of  factors  that  might  be  better  than  the  selected  six 
factors,  or  perhaps  more  likely,  there  is  just  a  great  deal  of 
variation  among  evaluators  in  their  rating  of  risk. 

4.4.4  Comparison  of  Metric-to-Risk  Conversion  Methods. 

a.  In  sections  4.4. 1.1  and  4. 4. 1.2,  two  methods  were  described 
whereby  the  general  supportabi 1 ity  metric  (rating)  could  be  converted 
to  risk.  The  first  of  these  methods  is  referred  to  as  a  simple 
linear  conversion  function,  the  second  as  a  linear  regression 
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approach.  Those  two  methods  were  compared  in  section  4. 4. 1.2,  spec¬ 
ifically  in  figure  4-4.  That  comparison  showed  that  the  linear 
regression  method  better  represents  the  data  collected.  Although 
application  of  the  linear  regression  method  is  somewhat  more  involved 
that  that  of  the  simple  linear  conversion  method,  the  calculations 
are  still  easy  enough  to  do  on  a  hand  calculator.  For  quick  approxi¬ 
mations  of  risk  from  the  genera!  supportability  metric,  the  linear 
regression  approach  may  be  used,  as  long  as  the  user  remains  aware  of 
the  error  potential  evident  in  the  scatter  of  data  about  the 
prediction  lines  shown  in  figures  4-3  and  4-4. 

b.  The  factor  regression  model  developed  in  section  4.4.3  is  the 
most  accurate  means  devised  here  for  predicting  risk  from  support- 
ability  metrics.  Figure  4-8  is  a  plot  of  transformed  risk  values 
(LRISK)  versus  the  rating  on  the  general  supportability  metric 
(ASUPPORT).  This  plot  is  basically  the  same  as  that  of  figure  4-3, 
with  the  straight  line  drawn  through  the  scatter  of  data  again 
representing  the  regression  on  the  general  supportability  metric. 
The  points  indicated  by  "0"  in  the  plot  are  actually  observed  data 
values,  those  indicated  by  "P"  are  predicted  from  the  regression  on 
the  supportability  factors,  and  those  indicated  by  an  asterisk  are 
coincidences  (to  within  the  resolution  of  the  plot)  between  observed 
and  predicted  transformed  risk  values.  From  the  plot,  it  is  clear 
that  the  factor  regression  model  better  represents  the  data  than  does 

the  general  supportability  metric  regression  model.  This  conclusion 

2 

is  confirmed  by  the  fact  that  the  R  value  for  the  factor  model  is 
higher  than  that  for  the  general  supportability  metric  model  (.44  as 
compared  to  .35). 

c.  A  drawback  to  the  model  using  supportability  factors,  however, 
is  that  its  application  is  computationally  intensive  and  therefore 
not  suited  to  a  hand  calculator.  Two  lengthy  steps  are  required  to 
use  that  model  in  predicting  risk.  First,  scores  for  each  of  the 
six  factors  must  be  computed  from  one  of  six  different  linear 
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functions  having  37  terms,  one  per  supportabi 1 ity  metric.  Second, 
the  factor  scores  must  be  substituted  into  the  regression  model  of 
section  4.4.3. 2  to  compute  transformed  risk.  The  last  step,  detrans- 
forming  the  transformed  risk  value  to  a  risk  value  is  the  same  for 
either  regression  model.  Although  this  procedure  is  too  tedious  for 
hand  calculation,  it  could  be  implemented  for  quick  and  easy  use  on  a 
computer. 

4.5  MAINTENANCE  DATA  ANALYSIS. 

a.  The  primary  objective  of  collecting  the  maintenance  activity 
data  is  to  generate  a  historical  data  base  from  which  baseline 
support  profiles  can  be  created.  Because  it  was  not  clearly  speci¬ 
fied  by  the  preliminary  methodology  (reference  8.3)  how  such  data 
would  be  used  to  create  the  required  profiles,  there  was  the  need  to 
establish  this  relationship.  Another  important  factor  to  be  inte¬ 
grated  is  "reality"  -  that  is,  what  is  really  possible? 

b.  Because  of  the  above  considerations,  the  reduction  of  the 
collected  data  to  some  most  common  denominator  became  a  primary 
concern  of  the  analysis.  It  was  necessary  to  integrate  the  data 
being  collected  at  each  site  and  determine  what  could  be  done  with 
reasonable  resources  such  as  would  be  available  during  a  normal 
system  OT&E,  and  which  data  was  likely  to  be  common  with  the  systems 
to  be  surveyed  in  the  future.  In  short,  the  survey  was  an  evolving, 
lessons  learned,  learning  curve  experience.  The  methodology  impact 
of  this  analysis  is  presented  in  section  VI,  including  an  extended 
example.  The  analysis  comments  presented  in  this  section  include 
data  items  of  most  interest,  availability  of  the  data,  consistency 
and  accuracy  of  the  data,  ways  to  improve  the  validity  of  the  data, 
regression  analysis  of  baseline  profile  factors,  and  general  conclu¬ 
sions/observations  . 
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4.5.1  Summary  Observations  on  Collected  Maintenance  Data. 

4. 5. 1.1  Software  Release  Data  Items  of  Interest.  Major  software 
maintenance  parameters  of  interest  include: 

(1)  Block  release  start  date,  engineering  completion  date, 
field  date 

(2)  Number  of  personnel  assigned  to  each  block  release  and 
percentage  of  the  time  these  personnel  are  dedicated  to 
the  release 

(3)  Skill  level  of  personnel  assigned  to  block  release 

(4)  Estimated  level  of  resource  requirements  (personnel  and 
systems)  for  block  release:  at  start  date 

(5)  Actual  level  of  resources  consumed  (personnel  and 

systems)  for  block  release:  at  engineering  completion 

date 

(6)  For  each  change  request  in  the  block  release,  the  type, 
complexity,  priority,  estimated  and  actual  resource 
requirements,  configuration  control  dates 

All  of  this  data  should  be  available  through  a  computerized,  con¬ 
figuration  management,  status  accounting  function.  No  system  had  all 
the  information  available  in  an  easily  accessible  form.  Appendix  D 
contains  a  list  of  the  systems  and  the  release  data  items  which  were 
collected.  Some  general  statistics  of  interest  pertaining  to  the 
maintenance .release  data  are  presented  in  table  4-5. 

4.5. 1.2  Avai labi 1 ity  of  Data. 

a.  Data  was  collected  from  nearly  all  possible  personnel  and 
organizational  "pieces"  of  a  support  organization.  The  formal 
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configuration  management  function  at  a  given  site  would  generally 
have  summary  data  on  each  block  release.  The  Air  Force  Form  75  is 
the  formal  mechanism  for  ALCs,  but  the  data  on  these  forms  has  a  wide 
flexibility  of  format  and  meaning.  Estimates  of  block  release  person 
effort  broken  into  maintenance  project,  configuration  management, 
test,  IV&V,  contractor,  and  overall  management  categories  is  avail¬ 
able.  Occasionally,  the  individual  list  of  change  requests  incorpor¬ 
ated  into  the  block  release  is  included.  A  schedule  for  the  various 
block  release  phases  is  supposed  to  be  attached.  No  information 

Table  4-5. 

Summary  of  Maintenance  Release  Data  Statistics 

SYSTEMS 

Total  Numb ar  -  91 

#  OFP  -  02 

#  £W  -  3 

#  C-E/C3I  -  10 

#  ATD/OFT  -  7 

#  ATE  -  7 

1*  SIM  -  6 

#  SUP  -  11 

BLOCK  RELEASES 

Total  Number  Reported  -  306 

Number  Use-ful  -for  Prof  i  lee  -  273 

Number  with  TYPE  Changes  -  008 

Number  with  COMPLEXITY  Changes  -  173 

Number  with  PRIORITY  Changes  -  311 


CHANGES  * 

Total  Number  Reported 
TYPE  Changes 


-  12797 


Total  Number  Reported 

- 

12769 

#  Correction* 

- 

99e2 

•  V  s 

79 

#  Enhancements 

- 

2733 

:  7.  =■ 

*■> 

#  Conversions 

- 

«  -M 

:  7.  = 

0 

COMPLEXITY  Changes 

Total  Number  Reported 

- 

7627 

#  High  Complexity 

- 

937 

:  7.  = 

i  1 

*  Medium  Complexity 

- 

2332 

:  7.  a 

#  Low  Complexity 

- 

3938 

:  7.  ■ 

■T  “> 

PRIORITY  Changes 

Total  Number  Reported 

- 

12799 

#  Emergency 

137 

:  7.  * 

I 

*  Urgent 

- 

2627 

•  *fm  S 

21 

#  Normal 

- 

9973 

2  V.  * 

78 

COMPUTATIONS  * 

Number  o-f  Changes  Per  Release 
Total  Available  Farion  Months 
Availao;®  Parson  Months  Per  Changi 


-  Minuteman  releases  (?)  ar j  not  included  in  jata 
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(other  than  the  naming  scheme  for  the  requests)  is  available  to 
stratify  the  change  requests.  The  lower  level  detail  such  as  type, 
complexity,  priority,  effort  estimates  and  control  dates  for  each 
change  request  is  usually  contained  in  working  engineering  notebooks. 
This  latter  information  is  not  consistently  recorded  nor  completely 
available  for  all  systems. 

b.  Actual  person  effort  for  a  block  release  was  almost  always 
unavailable.  The  complexity  of  each  change  request  was  not  directly 
available.  The  stratification  of  change  requests  by  the  type 
"conversion"  was  almost  always  unavailable,  despite  the  fact  that 
conversions  are  being  done  all  the  time,  and  are  frequently  very 
complex.  The  conversions  are  generally  included  with  enhancements 
for  the  maintenance  activity  data  collected.  The  personnel  resource 
data  was  obtained  through  subjective  estimates  by  the  support 
personnel.  Most  often  only  engineering  release  dates,  and  fielded 
dates  were  known.  The  release  start  date  is  usually  not  well-defined 
due  to  the  nature  of  processing  change  requests.  Where  a  standard 
cycle  (e.g.,  18  months)  for  block  releases  was  known,  the  start  date 
was  estimated  from  the  end  dates.  There  is  usually  a  long  time  (6  to 
13  months)  between  completion  of  the  maintenance  engineering  effort 
and  the  operational  fielding  of  the  software  release.  Hence,  the  end 
of  the  engineering  effort  (through  OT&E  phase  functions)  is  the  term¬ 
inology  used  for  the  engineering  completion  date.  The  start  date  is 
when  the  bulk  of  the  maintenance  block  release  effort  begins. 

c.  Oespite  the  limitations  of  the  available  data,  support 
personnel  were  frequently  able  to  reconstruct  some  of  the  release 
data  not  readily  available  such  as  complexity  level  (high,  medium, 
low),  important  release  dates,  and  personnel  resource  data.  As  an 
example,  the  Ogden  ALC  F - 16  and  F-4  support  personnel  were  able  to 
provide  complexity  estimates  to  each  change  request  in  each  release 
in  a  little  less  than  a  person  day  (several  personnel  working  a 
little  more  than  an  hour  apiece). 
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4. 5. 1.3  Consistency  and  Accuracy  of  the  Data. 


a.  Part  of  the  accuracy  and  consistency  problems  of  the  data 
reflect  the  lack  of  adherence  to  the  model  of  software  maintenance 
activity  as  described  in  section  3.2.  Many  systems  had  not  been 
formally  transferred  to  the  support  organization.  Consequently,  some 
data  such  as  development  contractor  support  data  was  not  available, 
but  some  organic  data  was  available.  This  has  contributed  to  data 
inconsistency.  The  non-uniform  support  across  organic,  developer, 
and  subcontractor  personnel  seemed  to  be  rather  prevalent  at  the 
ALCs,  even  for  software  systems  (e.g.,  B-52,  E-3A)  which  one  would 
suspect  have  already  been  transferred  and  are  under  full  ALC  support 
responsibi 1 ity. 

b.  The  available  person  time  was  adopted  for  use  in  the  develop¬ 
ment  of  baseline  profiles  because  no  other  personnel  effort  data  was 
uniformly • avai lable.  Available  person  time  for  a  block  release  is 
computed  as  the  product: 

APT  *  (N P)  (PDS)  (PDR)  (DBR) 


where:  APT 
N  P 
PDS 


PDR 

DBR 


Available  person  time  (months) 

Number  of  persons  assigned  to  software  system 
Percentage  of  the  persons  dedicated  to  the  software 
system  (versus  shared  time  with  other  software 
systems) 

Percentage  of  the  persons  dedicated  to  the  block 
release  (versus  other  block  releases  for  this  system) 
Engineering  duration  of  the  block  release  (months). 


c.  Use  of  available  person  time  leads  to  a  variance  in  the  data 
which  can  greatly  affect  the  shape  and  accuracy  of  the  profile 
curves.  For  example,  using  the  limited  amount  of  data  on  estimated 
and  actual  person  effort  available  for  block  releases  along  with 
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information  gathered  during  the  survey  interviews,  the  ratio  of  ^ 

actual  person  effort  in  a  block  release  to  available  person  time 

ranged  from  0.3  to  0.9.  Thus,  a  value  of  2.0  available  person  months 

per  change  for  a  given  release  might  reflect  a  range  in  actual  person 

months  per  change  of  0.6  up  to  1.8.  This  variance  confirms  that  ALC 

personnel  are  being  utilized  at  different  capacities  across  different 

releases,  perhaps  depending  upon  change  requirements.  It  may  also 

account  for  the  rather  wide  disparity  in  the  available  person  months 

per  change  across  releases  for  a  given  system  and  across  different 

systems.  It  also  indicates  that  a  large  amount  of  overhead  (effort 

not  directly  attributable  to  software  releases)  is  included  in  the 

available  person  time. 

d.  For  example,  in  one  release  personnel  could  be  utilized  at 
90  percent  capacity  to  produce  50  changes  over  a  12-month  engineering 
release  period.  In  another  release,  personnel  could  be  utilized  at 
30  percent  capacity  to  produce  21  changes  over  a  12-month  engineering 

release  period.  The  actual  productivity  would  be  the  same  in  these  * - 

two  cases.  The  available  person  time  per  change  would  be  signifi¬ 
cantly  different,  reflecting  the  variance  in  overhead. 

e.  These  observations  simply  support  normal  expectations.  The 
importance  of  the  data  is  to  provide  some  boundaries  and  guidelines 
upon  which  'some  better  decisions  can  be  made.  The  observation  of  the 
30  to  90  percent  utilization  variance  would  be  very  valuable  if  it 
could  be  validated.  The  data  for  the  available  person  time  is  some¬ 
what  inaccurate  since  it  was  not  always  possible  to  determine 
precisely  how  many  persons  were  assigned  to  a  given  release.  The 
general  assumption  that  personnel  are  assigned  in  some  uniformly 
dedicated  percentage  of  time  to  a  system  and  to  a  release  is 
obviously  not  correct,  except  as  a  first  approximation.  However, 
using  the  available  person  time  should  give  a  reasonable  upper  limit 
to  the  risk  estimation.  This  would  allow  for  some  tradeoffs  such  as 

assuming  a  larger  apparent  risk  in  anticipation  of  greater  utiliza-  y,. 

1  * 

tion  of  personnel.  *** 
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f.  Collected  data  items  which  were  very  accurate  (perhaps  within 
5  percent  error)  included  the  total  number  of  changes,  number  of 
corrections,  and  number  of  non-corrections  (enhancements  plus  conver¬ 
sions)  in  each  block  release.  An  interesting  observation  (from 
table  4-5)  is  that  the  percentage  of  .corrections  (73  percent)  is  much 
higher  than  the  percentage  of  non-corrections  (22  percent).  A  brief 
analysis  of  the  available  effort  data  seems  to  indicate  more  effort 
is  spent  on  an  average  enhancement  than  on  an  average  correction,  but 
not  enough  to  cause  the  percentage  of  effort  spent  on  enhancements  to 
be  more  than  on  corrections.  This  data  seems  to  contradict  the 
generally  accepted  results  from  the  Lientz-Swanson  research  (refer¬ 
ences  7.11  and  7.12)  and  several  other  research  efforts.  The 
research  results  indicate  that  the  support  effort  is  divided  across 
corrections,  enhancements,  and  conversions  at  the  respective  percent¬ 
ages  20  percent,  60  percent,  and  20  percent.  The  results  are 
primarily  based  upon  subjective  responses  from  ADP  software  support 
managers,  but  the  concept  of  "more  effort  is  spent  on  enhancements 
than  corrections  during  software  maintenance"  is  well  accepted  within 
the  military  software  support  community.  Because  of  the  lack  of 
overall  data  consistency,  no  certain  conclusions  can  be  made,  but 
this  observation  would  be  interesting  to  revisit  if  actual  effort 
data  by  change  request  were  collected  in  the  future  so  that  a 
statistical  analysis  could  be  performed. 

4. 5. 1.4  Techniques  to  Improve  Data  Validity.  There  are  many  tech¬ 
niques  to  improve  the  validity  of  data  such  as  has  been  gathered 
during  the  survey  visits  of  this  study.  The  primary  techniques  which 
have  been  used  during  the  evolution  of  the  survey  process  include: 

(1)  Delphi-feedback.  By  asking  ne  sources  of  data  for 

updated,  more  accurate  data  based  upon  better  terminology 
and  review  of  similar  data  from  other  sources,  the 
validity  of  the  data  is  improved. 
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(2)  Cal ibration.  More  careful  explanation  of  the  termi¬ 
nology,  use  of  the  data,  and  expected  relationships  of 
the  data  should  improve  the  validity  of  the  data. 

(3)  Evaluation  Time.  The  nature  of  the  survey  process  meant 
the  evaluators  were  greatly  constrained  in  evaluation 
time.  The  more  normal  evaluation  procedure  may  still  be 
constrained,  but  will  improve  the  validity  of  the  data. 

4. 5. 1.5  General  Concl us  ions /Observations . 

a.  The  bottom  line  is  whether  the  results  of  the  maintenance 
(release)  data  analysis  supports  and/or  improves  the  current  risk 
assessment  methodology.  The  analysis  of  the  data  indicates  that  the 
methodology  has  been  improved,  made  more  realistic,  and  is  now 
capable  of  an  actual  application  pilot  study  (see  chapter  VI). 


b.  The  maintenance  data  collection  must  be  standardized  and  a 
central  data  repository  established  before  very  much  accuracy  and 
consistency  of  the  data  can  be  realized.  Recommendations  on  how  this 
might  be  done  are  presented  in  section  3.6. 


c.  The  baseline  support  profile  agreement  between  user  and 
supporter  must  be  integrated  into  the  life  cycle  acquisition  process. 
This  is  important  to  the  success  of  the  RAMSS  concept. 

d.  Results  of  current  statistical  analysis  is  presented  in  the 
fol lowing  sections . 


4.5.2  An  Examination  of  Variables  Potentially  Associated  with  Avail¬ 


able  Person  Time. 


a.  With  the  maintenance  data  collected  for  this  effort,  available 
person  time  could  be  character i zed  in  either  of  two  ways:  as  person 


IV  -42 


THE  BDM  CORPORATION 


BDM/A-85-0510-TR 


time  £er  release  or  as  person  time  £§r  change,  where  a  change  is  the 
elemental  unit  of  software  modification  within  a  release  consisting 
of  multiple  changes.  Person  time  per  change  is  merely  the  total 
available  person  time  for  a  release  divided  by  the  total  number  of 
changes  comprising  the  release.  Use  of  person  time  per  change  offers 
the  advantage  that  it  tends  to  normalize  all  the  releases  to  a  common 
basis.  Comparison  of  releases  is  difficult  on  a  person  time  per 
release  basis,  because  the  number  of  changes  in  each  release  must  be 
accounted  for.  In  this  data  set,  the  number  of  changes  per  release 
varies  from  one  to  hundreds,  making  comparisons  between  releases  on  a 
person  time  per  release  basis  quite  complicated.  For  this  reason, 

person  time  per  change  was  chosen  as  the  variable  to  represent  avail¬ 
able  person  time  in  this  study.  The  units  of  time  used  throughout 
are  months,  so  that  the  variable,  person  time  per  change,  becomes 

person  months  per  change  (PMPC). 

b.  Along  with  available  person  time,  many  other  items  of  infor¬ 
mation  were  collected  on  software  systems  and  on  the  block  releases 
of  software  changes  associated  with  those  systems.  Two  key  pieces  of 
information  used  in  developing  the  maintenance  profiles  of  section  V 
are  the  type  of  software  system  and  the  site  at  which  the  system 
software  is  maintained.  In  section  V,  separate  maintenance  profiles 
(histograms  showing  the  frequencies  of  available  PMPC  values)  are 

given  for  each  site  and  software  system  type.  As  was  anticipated 

before  any  data  were  collected,  there  are  substantial  differences  in 
PMPC  among  the  various  software  types  and  sites;  hence,  software  type 
and  site  are  two  variables  that  are  apparently  associated  with  PMPC. 
Other  variables  thought  to  be  potentially  associated  with  PMPC  were: 

(1)  PTCORR  -  the  proportion  of  changes  of  correction  type 

(2)  PCIOW  -  the  proportion  of  changes  of  low  complexity 


(3)  PPNORM  -  the  proportion  of  changes  of  normal  priority 
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(4)  KLINES  -  the  number  of  K-lines  (thousands  of  lines)  of 
source  code  in  the  software  system  being  maintained 

(5)  PCHILEV  -  the  percentage  of  high-level  computer  languages 
(languages  ocher  than  assembly)  used  in  the  source  code 

(6)  AVGSKILL  -  the  average  skill  level  of  the  software  maint¬ 
enance  personnel  (derived  from  system  background  data 
collected  on  the  survey  forms--see  appendix  C). 

c.  Before  proceeding  with  an  analysis  of  the  variables  listed 
above,  it  is  appropriate  to  present  for  orientation  purposes  some 
summary  information  on  the  releases  for  which  data  was  obtained. 
Figure  4-9  is  a  chart  showing  the  counts  of  releases  for  which  data 
was  obtained  in  each  software  type  and  at  each  site.  These  counts 
reflect  all  the  data  records,  or  cases,  stored  in  the  dBASE  III  data¬ 
base  RLS_SMRY.DBF,  including  those  cases  that  will  be  discarded  for 
the  analysis  that  follows  because  they  lack  data  for  one  or  more  of 
the  variables  needed  or  are  otherwise  inappropriate  for  inclusion  in 
the  analysis.  The  software  type  and  site  abbreviations  are  defined 

in  tables  5-1  and  5-2  of  section  V.  Note  that  the  pattern  of  non¬ 
empty  cells  in  the  matrix  is  quite  sparse,  and  counts  within  the 

cells  vary  from  1  to  110.  Also,  the  marginal  counts  for  two  groups 
are  quite  low:  software  type  SIM  and  site  CASTLE  had  only  5  and 
6  releases,  respectively  From  the  counts  in  the  cells,  it  is  apparent 
that  almost  all  of  the  8  sites  are  dominated  by  releases  in  a  single 
type  of  software;  thus,  the  differences  in  PMPC  that  are  seen  among 
sites  in  the  maintenance  profiles  of  section  V  may  be  primarily 
attributed  to  differences  among  software  types.  For  this  reason,  the 
analysis  of  the  next  two  sections  will  use  software  type  as  a 

variable,  but  not  site. 
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Figure  4-9.  Counts  of  Releases  in  Raw  Data  by  Site 
and  Software  Type 


4. 5. 2.1  Analysis  Approach. 


a.  The  purpose  of  this  analysis  is  to  determine  whether  a 
proposed  mathematical  model  involving  variables  in  addition  to  soft¬ 
ware  type  can  be  used  to  predict  PMPC  more  effectively  than  does  a 
model  involving  only  software  type.  This  statement  presupposes  that 
software  type  is  useful  is  predicting  P^PC,  something  that  has  not 
yet  been  demonstrated  but  will  be  in  the  analysis.  The  additional 
variables  to  be  used  are  those  listed  above  in  section  4.5.2.  Should 
some  of  these  variables  prove  to  be  useful  in  predicting  PMPC,  then  a 
model  incorporating  them  could  be  exploited  to  provide  improved 
estimates  of  the  supportabi 1 i ty  risk  associated  with  a  particular 
software  system. 
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b.  The  proposed  model  is  a  linear  regression  model  of  the  form 

Y  3  bg  +  +  •••  +  +  +  • . .  +  at— 1+  s* 

where 

Y  *  the  logarithm  of  person  months  per  change  (PMPC), 

Xj  *  the  i-th  covariate  (one  of  the  6  variables  above), 

b-j  *  the  regression  coefficient  for  the  i-th  covariate, 

aj  =  an  indicator  variable  for  the  j-th  software  type, 

t  3  the  number  of  different  software  types, 

e  2  a  random  component. 

The  logarithm  of  PMPC  is  used  as  the  dependent  variable,  Y,  in  this 
model,  because  preliminary  examinations  of  the  data  indicated  differ¬ 
ences  in  the  variance  of  PMPC  between  the  software  types--differences 
of  a  nature  that  the  logarithm  transformation  is  known  to  alleviate 
in  many  situations.  There  are  t  -  1  software-type  indicator  vari¬ 
ables  in  this  model  instead  of  t,  because  the  intercept  bg  accounts 
for  the  t-th  software  type. 

c.  This  model  is  an  analysis  of  covariance  type  model  in 
regression  form.  The  idea  underlying  the  model  is  that  a  given  soft¬ 
ware  type  has  an  average  (mean)  PMPC  value  that,  in  general,  is 
different  from  those  of  the  other  software  types.  In  addition,  the 
covariates  contribute  to  differences  in  PMPC,  so  that  two  systems  of 
a  given  software  type  but  of  different  sizes  (KLINES),  for  example, 
will  be  expected  to  have  different  predicted  PMPC  values. 

4. 5. 2. 2  Results. 


a.  In  fitting  the  regression  model  of  section  4. 5. 2.1,  not  all  of 
the  cases  of  data  that  were  collected  were  used.  Of  the  336  total 
cases  collected,  29  were  for  releases  having  only  one  software 
change;  they  were  omitted  because  such  releases  are  atypical  and  do 
not  conform  to  the  concept  of  releases  made  up  of  multiple  changes. 
Another  63  cases  were  omitted  because  they  were  missing  data  for 
variables  required  in  the  analysis.-  Finally,  all  5  cases  for  the 
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software  type  SIM  were  omitted  due  to  strong  indications  of 

irregularities  in  the  data.  The  remaining  239  complete  cases  were 
used  in  the  analysis. 

b.  Results  for  the  full  regression  model  of  the  preceding  section 

appear  in  figure  4-10.  The  six  software  types  (t  =  6  in  the  model, 
since  the  type  SIM  was  left  out)  collectively  account  for  a  signifi¬ 
cant  portion  of  the  variation  in  Y  (this  was  ascertained  in  an 
analysis  of  variance  table  not  shown  here).  Of  the  six  covariates, 
though,  only  one,  PTCORR,  is  shown  to  have  a  regression  coefficient 
significantly  different  from  zero,  as  evidenced  by  the  significance 
probability  of  0.0036  under  the  column  heading  "P(2  TAIL)"  in  the 

table  at  the  bottom  of  figure  4-10.  However,  even  this  significant 
result  for  a  covariate  is  questionable,  since  one  of  the  diagnostic 

plots  investigated  for  these  results  shows  rather  clearly  that  an 
increase  in  the  variance,  not  a  decrease  in  the  mean,  of  Y  is 
probably  creating  a  spurious  significant  result.  Further  analysis 
could  be  done  to  check  this  assertion. 

2 

c.  The  R  statistic  for  this  model  is  a  very  low  .2656,  implying 

that  the  model  is  unable  to  explain  the  majority  of  the  variation  of 
Y  about  its  mean.  This  situation  may  be  attributed  largely  to  the 
coarse  nature  of  the  data.  While  considerable  care  was  taken  in 
collecting  and  processing  the  data,  no  amount  of  care  could  change 

that  fact  that,  in  most  cases,  data  of  the  form  desired  for  this 
study  simply  did  not  exist,  and  coarser  surrogate  data  had  to  be 
used.  No  doubt  further  analysis  might  yield  greater  insights  into 
this  data,  but  for  now  there  is  little  evidence  that  any  of  the 
covariates  examined  here  have  a  consistent  bearing  on  PMPC. 

d.  The  above-mentioned  statistical  significance  of  the  software 
types  implies  that  at  least  two  of  the  software  types  have  signifi¬ 
cantly  different  mean  values  of  Y.  Since  the  logarithm  transform¬ 
ation  used  to  get  Y  is  a  monotone  function,  the  inference  may  also  be 
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Figure  4-10.  Results  for  Regression  Model  Involving  Software 
Types  and  Covariates 
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made  that  at  least  two  of  the  software  types  differ  in  PMPC. 
Therefore,  it  is  advisable  when  using  the  maintenance  profiles  of 
section  V  to  estimate  risk  that  the  separate  profiles  for  the 
software  types  be  used  instead  of  the  all-inclusive  profile  in  which 
data  for  all  the  software  types  are  lumped  together. 
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SECTION  V 


MAINTENANCE  PROFILES 

5.1  GENERAL  DESCRIPTION  OF  MAINTENANCE  PROFILES. 

a.  The  parameters  plotted  for  the  historical  maintenance  profiles 
are  the  available  person-months  per  change  on  the  x-axis  and  the 
number  of  releases  on  the  y-axis.  The  x-axis  consists  of  discrete 
intervals.  The  resulting  plot  thus  is  a  frequency  histogram  with 
each  rectangular  "box"  representing  the  number  of  releases  for  which 
the  available  person-months  per  change  fell  within  the  discrete 
interval  on  the  x-axis  at  the  base  of  the  "box."  Figure  5-1  shows  a 
generic  example  of  a  maintenance  profile. 


AVAILABLE  PERSON-MONTHS 
PER  CHANGE 


Figure  5-1.  Generic  Maintenance  Profile  (Histogram)  for 
Available  Person-Months  per  Change 

b.  Available  person-months  per  change  is  defined  in  the  glossary 
and  explained  in  more  detail  in  section  4.5.  There  are  a  variety  of 
other  parameters  which  could  constitute  the  x-axis  parameter  for 
maintenance  profiles,  but  this  one  was  chosen  for  reasons  explained 
in  section  4.5.2. 

c.  All  historical  maintenance  orofiles  contained  in  the  remainder 
of  this  section  were  produced  by  the  3MDP  statistical  analysis 
program  resident  on  a  VAX  11/730  computer. 
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5.2  MAINTENANCE  PROFILES:  ALL  SYSTEMS. 

a.  A  historical  maintenance  profile  for  block  re. 'eases  over  all 
systems  for  which  the  necessary  data  were  collected  is  shown  in 
figure  5-2.  The  histogram  in  figure  5-2  is  oriented  somewhat  dif¬ 
ferently  than  that  in  figure  5-1.  It  has  been  rotated  clockwise  by 
90  degrees  so  that  the  available  person-months  per  change  (PMPCHNG) 
interval  values  appear  on  the  vertical  axis  and  the  frequency,  or 
number,  of  releases  appears  on  the  horizontal  axis.  Each  "X"  in  the 
plot  represents  1  release  (referred  to  as  an  "observation"  at  the  top 
of  the  plot).  The  numerical  values  listed  along  the  vertical  axis 
under  "INTERVAL  NAME"  are  upper  limits  of  the  PMPCHNG  intervals.  For 
example,  36  releases  had  PMPCHNG  values  greater  than  0  and  less  than 
or  equal  to  1.  The  four  columns  of  numbers  at  the  right  of  the 
histogram  are,  respectively,  frequencies  for  the  intervals,  cumula¬ 
tive  frequencies,  percentages  of  the  total  release  count  for  the 
intervals,  and  cumulative  percentages. 


I 
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b.  At  the  top  of  figure  5-2  are  printed  three  statistics  of 
interest.  They  indicate  that  there  were  280  releases  (of  a  total  of 
336  for  which  some  data  were  gathered)  for  which  sufficient  data  were 
obtained  to  allow  calculation  of  available  person-months  per  change 
and  which  had  more  than  one  change.  Releases  having  only  one  change 
were  judged  to  be  inconsistent  with  the  idea  of  a  release  having 
multiple  changes;  furthermore,  many  of  the  s ingle-change  releases 
were  found  to  be  extraordinary  in  several  respects  from  the  bulk  of 
multiple-change  releases.  For  these  280  releases,  the  overall  mean 
(average)  available  person-months  per  change  was  3.853  with  a 
standard  deviation  of  3.585. 

c.  To  further  exemplify  the  use  of  the  profile,  note  that 
65  releases  (23.2  percent  of  the  280)  had  PMPCHNG  values  greater  than 
1  and  less  than  or  equal  to  2,  and  101  releases  (36.1  percent)  had 
values  less  than  or  equal  to  2.  Only  6.4  percent.  (100  percent  minus 
93.6  percent)  of  the  releases  had  PMPCHNG  values  exceeding  10. 
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5.3  MAINTENANCE  PROFILES:  BY  SITE. 

a.  Maintenance  data  were  collected  by  block  release  from  each  of 
eight  sites;  these  sites  are  tabulated  with  abbreviations  in 
table  5-1. 


Table  5-1. 

Sites  From  Which  Maintenance  Data  Were  Collected 


SITE 

AB8R 

Castle  AF3 

CASTLE 

NORAD/SPACE COM 

NORAD 

Oklahoma  City  ALC 

OC-ALC 

Ogden  ALC 

00-ALC 

Sacramento  ALC 

SM-ALC 

Tinker  AFB 

TINKER 

Warner-Robbins  ALC 

WR-ALC 

Langley  AFB 

LANGLEY 

In  this  section,  the  same  data  that  were  presented  in  section  5.2  are 
grouped  separately  by  site  to  allow  comparisons  between  sites. 
Figure  5-3  is  a  side-by-side  presentation  of  the  maintenance  profiles 
(histograms)  for  each  of  the  sites.  The  histograms  differ  from  the 
one  in  figure  5-2  chiefly  in  that  the  PMPCHNG  interval  values  along 
the  vertical  axis  decrease  (rather  than  increase)  from  top  to  bottom, 
and  the  values  themselves  are  the  midpoints  (instead  of  the  upper 
limits)  of  the  PMPCHNG  intervals.  Site  abbreviations  are  listed  near 
the  top  of  the  figure. 

b.  Within  the  histograms,  each  asterisk  represents  1  release. 
For  some  intervals,  the  number  of  asterisks  exceeds  the  allowable 
width  of  the  histogram,  so  the  total  frequency  for  the  interval  is 
printed  at  the  end  of  the  asterisk  row.  For  example,  *in  ,the  NORAD 
histogram  there  were  17  releases  for  the  interval  having  a  midpoint 
of  2.  The  lower  and  upper  limits  of  that  interval  are  1.5  and  2.5, 
respect i vely.  Each  interval  includes  its  upper  limit  (but  not  its 
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lower),  so  for  NORAD,  17  releases  had  PMPCHNG  values  greater  than  1.5 
and  less  than  or  equal  to  2.5  (see  for  comparison  figure  5-4).  (Note 
that  3MDP  forces  a  midpoint  of  0.000  to  mean  an  interval  <  0.5  and 
>-  0.5.  Because  there  are  no  negative  values,  this  interval  is 
actually  <_  0.5  and  >  0  for  this  data.) 

c.  Just  underneath  each  histogram  are  printed  several  statistics 
of  interest,  most  notably  the  mean,  the  standard  deviation 
(STD.DEV.),  and  the  minimum  and  maximum  PMPCHNG  values  for  the 
releases  of  each  site.  Also  listed  is  tne  total  number  of  releases 
(SAMPLE  SIZE)  included  in  the  histogram  for  each  site.  At  the  bottom 
of  the  figure  are  the  same  statistics  for  all  sites  combined--tnese 
agree  with  the  statistics  shown  at  the  top  of  figure  5-2. 

d.  Figures  5-4  through  5-11  are  the  site  maintenance  profiles  in 
the  same  format  as  that  of  figure  5-2.  The  abbreviation  for  the  site 
to  which  the  profile  applies  is  printed  on  the  third  line  at  the  top 
of  each  figure. 

5.4  MAINTENANCE  PROFILES:  BY  SOFTWARE  SYSTEM  TYPE. 

a.  Seven  software  system  types  were  represented  among  the  soft¬ 
ware  maintenance  data  collected  across  all  sites.  Table  5-2  is  a 
list  of  the  software  system  types  and  their  abbreviations/acronyms. 

b.  Separate  maintenance  profiles  for  the  software  system  types 
are  depicted  side  by  side  in  figure  5-12  in  a  format  identical  to 
that  of  figure  5-3.  System  type  abbreviations  appear  near  the  top  of 
the  figure.  The  maintenance  profiles  by  software  system  type,  in  the 
format  of  figures  5-2  and  5-4  tnrough  5-11,  are  presented  in 
figures  5-13  through  5-19,  with  the  system  type  abbreviation  given 
above  each  plot. 
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Automatic  Test  Equipment 
Communications -Electronics 
Electronic  Warfare 
Operational  Flight  Program 
Aircrew  Training  Device 
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Figure  5-10.  Maintenance  Profile  for  Tinker  AFB 


THE  BDM  CORPORATION 


BDM/A-85-0510-TR 


tu  •  rjau^.-.aDOO*— rwr-*-auaDccaoaoajaocoau©oooooo 

OX  . . . . . . 

►—  o  -*«*'4jr^a>auco<r>aDO'0'0'0'Ocrcrcro'C7'a'0'U'cro'Oo©©ooo 


O  •  rs4'0'0f—  ^'^JO'^JOrvjfN.O^vJOO^JOOOOOOOO'XlOOOOOO 

QC  >-  . . . 

UJ  2  U"'rvicy*a}a3<N.©r*©r\*rsj0<‘'u©©<''i©0000©0©«"u0©0©©© 

a.  p.  ivi  m 

>■  • 

OX  r-(N«ini7<oo^l-(\j^(n^^^\Aini/>i/>inininirtin'0>0'0'0^^'0 

Z  O  <n  ^  ro  i'O  s/  sf  sf  v/  ^  ^  vi  ^  -j  ^ 

UJ  o 
Z> 

a  • 

UJ  H  N  O'  ©  **■.  O  »-*  ©  •— 1  Q  O  o  ©  ©  O  O  O  O  O  O  O  ©  O  O  © 

a  2  h 


<7*  © 

«—  ^  o 

•  NO  2 

>  •  o 

UJ  >X  M 

CJ  >—  l/> 


2  su 

z>  >*■  *- 

o  2 


O  (X 

<D  Uj 

x  r  oc 


X 

X 

X 

X 

X 

X 

X  x 
x  X 
xxr 
xxx 

XXX 
X  X  X  x  X 

X  X  X  x  x 

X  X  X  x  x 

X  X  x  X  X  X 


-J  OOOOOOOOOOOOOOOOOOOOOOOOOOOOOO 

©o©c  ©OOOOOOOOOOOOOOOOOOOOOOOOO 

>  oooooooo©o©ooooooooo©ooooo©©oo 

£X  0000000©00000000©000000©00©00©h- 

^xjuj  OOOOOOOO©  .  .  .  .  . . .  .  .  .  .  .  ^ 

K  X  •  •  •  •  •  •  •  •  »o-'<,M<T''iuA'0N(j,fl'O^'Nf^^iriuir-a0|i'O^ 

X  **  ^  ^  'J  U>  >4}  P*.  J ,  lj<  _  M.  M  «—*  on  —<  _•  t~*  '  \J  J  f\(  fj  'Vj  <\(  fNJ  _J 


y.vM* 


THE  BDM  CORPORATION 


BDM/A-85-0510-TR 


oj  •  r-r^moooooooooooooooooooooooooooo 

O  X  . . . . . . .  • 

^o^omoooooooooooooooooooooooooooo 
VJ  -*no«x>oooo©ooooooooo©ooooooooooooo 

OJ 

o  «  f-Of—  r^OOOOOOOOOOOOOOOOOOOOOOOOOOO 

oc  *-  J 

lu  z  vo©vO>oooooo©oooaooo©ooooo©©©ooooo 

a.  *— •  ^  in  «■* 

^  • 

ij  j  vO  'O'O'O'C'O'C'O'^  *0  'O'OO'O'O'O'O'O'O'O 

Z  3 

UU  VJ 

3 

d  • 

uu  t-  ^m^^ooooooooooooooooooooooooooo 


JO 

o  <-o 
•  o  Z 
>  •  o 


^  <o  ^ 
3  »- 

o  z 


o  o 

z  © 

X  X 


QC 

o 

o 

- 


oooooooooooooooooooooooooooo©© 

OOOCOOOOOOOOOOOOOOOOOOOOOOOOOO 

oooooooooooooooooooooooooooo©© 
oooooooooooooooooooooooooooooo^- 
ooooooooo  •  •  •  •  •  •  •  •  .  •  •  •  •  •  *  •  •  •  •  • 

*  •  *  •  •  •  •  • 

*-  <N  f'l  •sfvi-''Or—  iX>(7'  •  «wH^^,rvj(Njr\jrgN^,N'N|(\jr\j'r>j 


Figure  5-13.  Maintenance  Profile  for  ATD 
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Figure  5-18.  Maintenance  Profile  for  SIM 
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SECTION  VI 

METHODOLOGY  REVIEW 


6.1  BACKGROUND. 

a.  The  proposed  methodology  for  risk  assessment  of  software  sup- 
portability  was  first  presented  in  the  reference  8.3  report.  An 
update  of  the  methodology  illustrating  the  hypothetical  generation 
and  use  of  baseline  historical  maintenance  profiles  is  presented  in 
reference  3.9.  The  methodology  is  referred  to  as  the  Risk  Assessment 
Methodology  for  Software  Supportabi 1 ity  (RAMSS).  Further  development 
of  the  concepts,  primarily  in  the  form  of  an  extended  example  is  pre¬ 
sented  in  reference  8.6.  A  summary  of  the  impact  of  the  data  col¬ 
lected  upon  the  RAMSS  is  presented  in  table  4-1  of  this  report. 

b.  For  the  purpose  of  completeness,  a  brief  review  of  the  RAMSS 

major  features,  the  evaluation  procedure,  and  the  extended  example 
from  reference  8.6  is  presented  in  this  section.  The  analysis 

results  from  section  IV  and  use  of  actual  historical  maintenance  pro¬ 
files  from  section  V  would  provide  only  a  greater  level  of  detail  and 
accuracy.  Hence,  the  original  hypothesized  data  from  reference  8.6 
is  retained  as  part  of  the  extended  example. 

6.2  MAJOR  FEATURES. 

The  major  features  of  the  RAMSS  framework  include: 

(1)  A  baseline  software  supportabi 1 i ty  profile  of  expected 
software  maintenance  actions  is  proposed.  This  baseline 

would  evolve  as  necessary  into  an  agreement  between  the 
using  and  supporting  organizations.  It  would  be  derived 
using  a  historical  maintenance  data  base  as  a  guideline. 
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(2)  Software  supportabi 1 ity  evaluation  metrics  are  derived 
from  characteristics  of  the  software  products,  software 
support  environment,  and  the  software  life  cycle  process 
management.  The  evaluation  is  conducted  relative  to  the 
baseline  software  suoportabi 1 ity  profile. 

(3)  Estimated  software  supportabi 1 ity  risk  is  computed  from 
the  software  suoportabi lity  evaluation  metrics  using  a 
simple  conversion  function.  Iteration  with  feature  2 
above  will  eventually  result  in  an  estimated  risk  which 
is  acceptable  (i.e.,  the  baseline  acceptable  risk).  The 
conversion  function  (or  perhaps  a  more  empirical  func¬ 
tion)  must  be  validated  using  the  historical  maintenance 
data.  More  valid  methods  are  presented  in  section  IV  but 
do  not  influence  the  general  use  of  such  an  acceptable 
risk  value. 

(4)  Software  supportabi 1 ity  risk  is  evaluated  for  conse¬ 
quences  and  alternative  choices  by  direct  comparison  of 
the  baseline  acceptable  risk,  the  evaluated  risk,  the 
baseline  support  profiles,  and  the  evaluated  software 
supportabi 1 ity  characteristics. 

(5)  Elements  of  the  framework  can  be  applied  throughout  the 
software  development  and  support  life  cycle  phases  when¬ 
ever  software  evaluations  can  be  conducted. 

(6)  Elements  of  the  framework  are  applicable  to  any  software 
quality  evaluation  where  the  quality  affects  the  opera¬ 
tion  or  support  of  a  system.  As  an  example,  risk  assess¬ 
ment  of  software  reliability  could  use  a  similar  frame¬ 
work  with  a  baseline  of  operational  requirements  against 
which  software  reliability  is  evaluated. 
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6.3  EVALUATION  PROCEDURES. 

6.3.1  Life  Cycle  Phases.  Evaluation  of  software  supportability  is  a 
life  cycle  process.  There  are  key  points  (such  as  milestones  1,  2, 
3,  critical  design  review,  IOC,  PMRT)  throughout  a  software  system's 
life  cycle  where  application  of  a  RAMSS  (or  some  part  of  it)  would  be 
beneficial.  Benefits  which  might  occur  include:  early  planning  and 
trade-off  studies  for  software  support  facility  resource  require¬ 
ments;  early  view  of  potential  software  support  management  problems; 
early  visibility  of  user  requirements  fcr  expected  software  support 
actions;  capability  to  trace  software  supportability  risk  profile 
(i.e.,  measures  of  risk)  throughout  the  life  cycle;  early  view  of 
expected  software  supportability  risk  drivers;  and  the  actual 
assessment  of  the  risk  to  user  and  supporter  which  must  be  accepted 
before  support  of  the  software  can  be  assumed. 

6-3.2  Steps  of  Evaluation  Procedure. 

a.  The  basic  steps  of  a  procedure  to  control  the  evaluation  pro¬ 
cess  include:  planning;  evaluation;  analysis;  and  reporting.  Typi¬ 
cal  actions  for  each  of  these  steps  are  presented  in  figure  6-1. 

b.  From  the  perspective  of  a  RAMSS,  the  primary  function  in  the 
planning  phase  is  to  establish  an  appropriate  baseline  profile  of 
expected  maintenance  actions.  Because  of  the  level  at  which  the 
evaluation  is  being  conducted,  it  may  not  be  necessary  to  consider  a 
full  baseline  profile.  A  more  complete  methodology  should  establish 
guidelines  for  collecting  and  tailoring  the  base! ine  prof i le  data  to 
requirements  appropriate  for  the  desired  level  of  evaluation.  Tai¬ 
loring  the  data  might  involve  averaging  the  data  into  a  single  base¬ 
line  value  with  a  specified  range  of  variance.  This  would  greatly 
simplify  the  effort  of  the  evaluator,  but  would  also  add  uncertainty 
in  the  accuracy  of  the  evaluation  metrics. 
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c.  Conducting  the  evaluation  may  occur  over  a  short  or  long 
period  of  time  depending  upon  the  level  of  evaluation  being  con¬ 
ducted.  All  members  of  the  evaluation  team  (test  planners,  test  man¬ 
agers,  evaluators,  analysts)  should  be  cognizant  of  the  evaluation 
Drocess  and  calibration  requirements  for  the  evaluation.  It  is  this 
calibration  which  reduces  the  direct  misunderstanding  of  what  is  to 
be  evaluated,  reduces  the  subjectiveness  of  the  evaluation  questions 
and  resoonses,  and  improves  the  evaluation  accuracy  and  reliability. 

d.  The  current  evaluation  hierarchy  is  illustrated  in  figure  6-2. 
Most  of  the  application  of  a  RAMSS  will  be  during  the  analysis  of 
evaluation  results.  First,  the  evaluation  metrics  must  be  converted 
to  risk  measures.  The  conversion  can  occur  at  each  level  in  the 
evaluation  hierarchy  and  represents  the  relative  risk  contribution  of 
each  component.  This  process  is  illustrated  in  figure  6-3. 

e.  Since  each  evaluation  question  response  (as  the  average  of  all 
evaluator  responses)  has  a  variance,  there  is  an  associated  variance 
in  the  risk.  This  variance  determines  a  confidence  range  about  the 
evaluated  software  supportabi 1 ity  risk. 

f.  The  baseline  probability  density  function  derived  from  actual 
maintenance  data  (and  perhaps  some  heuristics)  is  used  to  provide 
insight  into  how  the  evaluated  risk  might  be  distributed.  This  pro¬ 
vides  a  perspective  on  the  magnitude  of  the  consequence  of  the  com¬ 
puted  risk. 

g.  From  the  evaluated  measures  of  risk  and  the  empirical  risk 
probability  density  functions,  it  is  possible  to  perform  simple 
tradeoff  studies  and  sensitivity  analysis.  An  extended  example  in 
section  6.4  illustrates  this  analysis.  The  possible  tradeoffs  to 
reduce  risk  by  improving  software  supportabi 1 i ty  or  modifying  the 
baseline  against  which  risk  is  determined  are  easy  to  explain  and 
ideal  for  inclusion  in  reports  to  decision  makers. 
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6.4  AN  EXTENDED  EXAMPLE  OF  THE  METHODOLOGY  APPLICATION. 

This  section  contains  a  hypothetical  example  of  applying  the 
RAMSS  framework  to  an  application.  The  historical  profiles  used  are 
representative  of  the  data  collected,  but  are  not  the  actual  oro- 
f i les. 

6.4.1  Terminology  and  Foundation. 

a.  Risk  is  the  potential  for  realization  of  unwanted,  negative 
consequences  of  an  event  (reference  8.10).  Risk  assessment  focuses 
upon  a  means  to  present  that  "potential",  primarily  as  a  probability. 
Determining  the  probability  across  possible  negative  consequences  of 
an  event,  and  across  applicable  events,  results  in  a  family  of  proba¬ 
bility  density  functions.  The  focus  of  risk  assessment  methodologies 
is  upon  the  derivation  of  a  baseline  probability  density  function 
representative  of  the  general  risk  function.  Then,  risk  is  defined 
when  a  measured  or  predicted  outcome  value  is  compared  to  the  base¬ 
line  density  function.  That  is,  risk  is  defined  by  those  outcomes 
(and  their  probabilities  of  occurrence)  which  are  negative  conse¬ 
quences  with  respect  to  the  baseline.  The  consequence  of  the  risk 
depends  upon  the  impact  of  the  negative  events  from  which  the  risk  is 
determined. 

b.  Software  supportabi 1 ity  is  a  measure  of  the  adequacy  of  per¬ 
sonnel,  resources,  and  procedures  to  facilitate  the  support  activi¬ 
ties  of  modifying  and  installing  software,  establishing  an  opera¬ 
tional  software  baseline,  and  meeting  user  requirements.  Negative 
outcomes  are  the  result  of  inadequacy  in  personnel,  resources  or  pro¬ 
cedures  to  accomplish  the  above  three  support  activities  in  an 
acceptable  manner.  "Acceptable"  is  defined  relative  to.  the  risk 
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agent's  acceptance  utility  and  the  baseline  probability  density  func¬ 
tion  of  expected  maintenance  activity.  Thus,  a  required  maintenance 
action  is  not  necessarily  negative.  Too  many  required  maintenance 
actions  or  the  inability  to  complete  a  required  maintenance  action 
within  a  specified  period  of  time  may  be  negative  depending  upon  the 
accepted  baseline. 


6.4.2  Historical  Maintenance  Profiles:  Example.  Suppose  the  his¬ 
torical  maintenance  profiles  for  all  systems  and  for  all  EW  systems 
are  as  shown  in  figure  6-4.  In  this  case,  300  block  releases  of 
which  50  are  for  EW  systems  have  been  analyzed.  The  available  person 
months  per  change  is  distributed  across  the  releases  as  shown  in  the 
figure.  Because  the  distributions  represent  a  frequency  count  they 
are  also  probability  density  functions.  The  risk  to  be  estimated, 
computed,  reduced,  and  so  forth  will  be  based  on  the  simplistic  idea 
that  the  more  person-months  per  change  allocated  to  accomplish  a 
block  release,  the  less  risk  is  involved  in  completing  the  block 
release  with  available  resources  (personnel,  systems,  etc.).  If  more 
person-months  per  change  is  required  than  allocated  by  the  user/sup¬ 
port  agreement,  then  there  is  unacceptable  risk. 


ALL  SYSTEMS  (300  RELEASES) 


EW  SYSTEMS  (50  RELEASES) 


» 

RELEASES 


available  PERSON  time  ;mo| 
?ER  CHANGE 


AVAILABLE  AEHSCN  TIME  ,Moi 
AER  Change 


Figure  6-4.  Historical  Maintenance  Profiles:  Example 
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6.4.3  Baseline  Agreement:  Example.  In  the  process  of  developing  a 
new  EW  software  system,  the  user  (HQ  TAC)  and  the  supporter  (Warner 
Robins  ALC)  need  to  arrive  at  an  agreement  as  to  the  expected  support 
requirements  for  the  new  system.  These  requirements  and  the  plan 
develooed  to  satisfy  those  ’■'equirements  are  specif;ed  ,-n  the  Computer 
Resources  Integrated  Support  Plan  (CRISP)  and  the  Operational/Support 
Configuration  Management  Plan  (0/SCMP).  DT&E  and  OT&E  organizations 
develop  specific  test  strategies  as  part  of  a  Test  and  Evaluation 
Master  Plan  (TEMP)  and  other  organization-specific  documents.  A  part 
of  the  support  requirements  should  be  based  upon  a  user/supporter 
agreement  on  the  baseline  software  support  profile.  For  this  exam¬ 
ple,  a  simplified  agreement  is  presented  in  figure  6-5. 

6.4.4  Baseline  Support  Profile  Risk  Computation:  Example.  The 
user/supporter  agreement  of  figure  6-5  allows  the  following  computa¬ 
tion  of  expected  available  person-months  per  change  (see  section 
4. 5. 1.3  for  more  discussion  of  this  computation).  The  computed  value 
of  2.67  is  then  plotted  on  figure  6-4  (as  a  vertical  dotted  line). 
The  estimated  risk  is  simply  the  area  under  the  curve  of  figure  6-4 
to  the  right  of  the  vertical  dotted  line.  The  corresponding  risk 
value  for  all  systems  and  for  EW  systems  only  is  shown  in  figure  6-6. 
This  estimated  risk  represents  the  acceptable  risk  to  the  user  and 
supporter  (since  the  baseline  support  profile  and  support  concept 
from  which  the  risk  is  derived  are  acceptable)  to  accomplish  tne 
indicated  block  release  with  the  specified  profile  of  change 
requests.  The  computation  of  the  risk  as  the  "integration"  over  the 
risk  probability  density  function  is  illustrated  in  figure  6-7. 

6.4.5  Evaluating  the  Software  Supportabi 1 i ty  Risk:  Example. 

a.  The  evaluation  of  software  supportabi 1 i ty  factors  (as  illus¬ 
trated  in  figures  6-2  and  6-3  in  section  6.3)  is  the  next  step  in  the 
RAMSS  process.  Parts  of  this  evaluation  can  be  conducted  throughout 
the  software  system  acquisition  life  cycle  as  is  appropriate.  Tie 
focus  of  this  example  is  tne  AFOTtC  OT&E  evaluation  at  or  near  the 
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AGREEMENT 

BLOCK  AVAILABLE  PERSON  TIME  PER  CHANGE  RISK 

•  »  MONTHS*  #  PERSONS  *',iQED  "  OVERLAP  FACTOR/#  CHG'S  '  ALL  EW 


1  *  (12  *  5  *  .80  *  .831/  15  =  2.67  .04  .10 


2 

»  (12  *  5  *  .80  *  ,67)/20  a 

1.61 

.39 

.61 

3 

■  (12  *  5  *  .80  *  ,67)/10  = 

3.22 

.03 

.06 

4 

*  (12  *  5  •  .80  *  .67J/25  « 

1.29 

.61 

.77 

S 

a  (12  *  S  *  .80  *  .67)/1 5  a 

2.14 

.17 

.28 

Figure  6-6.  Baseline  Support  Profile  Risk  Computation  Example  (1) 


•IlSK  COMPUTATION 

CD 

*!$*.«  - 


>  — 1  I  BISK,  =  14.300  =  04.  I 

«  *  1  ■  -  —1 


RISK,  ^  0  39 


EAPPC,  •  ESTIMATEO  AVAILABLE  PERSON 

time  per  change  eor  slock  release  i 
PU)  historical  discrete  probability 
oisr  EUNCnoN 


RISK)  a  0  03 
RISK,  *  0  6) 
RISK*  Z  0  17 


RISK,  a  5/50  =  0  10. 


RISK)  a  0  61 
RISK)  a  Q  06 
RISK,  *  0  77 
RISK*  a  0  28 


Figure  6-7.  Baseline  Support  Profile  Risk  Computation  Example  (2) 


end  of  the  software  system  full  scale  development  phase  prior  to  ini¬ 
tial  operational  capability. 

b.  The  results  of  the  hypothetical  evaluation  are  presented  in 
figure  6-3.  The  supportabi 1 ity  metric  conversion  to  risk  and  some 
potential  drivers  of  supportabi 1 ity  risk  are  also  shown  in  fig¬ 
ure  6-3.  The  evaluation  is  relative  to  the  block  release  1.  Note 
the  hypothesized  non-linear  model  of  the  supportabi 1 ity  metric  to 
risk  conversion.  The  actual  conversion  equation  is  probably  shaped 
more  like  a  hysteresis  curve  with  a  very  moderate  slope  at  *the  end 
points  and  a  very  rapid  slope  between  3.0  and  5.0.  A  cubic  equation 
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was  used  for  this  example,  but  section  IV  describes  better 
statistical  regression  derived  alternatives. 

c.  Henceforth  in  this  discussion,  the  risk  computed  as  a  result 
of  this  evaluation  will  be  called  the  "measured"  risk. 

6.4.6  Integration  of  Acceptable  and  Measured  Risk:  Example.  Using 
the  acceptable  risk  and  the  historical  maintenance  profiles,  the  next 
step  is  to  determine  what  the  required  productivity  (available  person 
months  per  change)  would  be  on  the  basis  of  the  measured  risk.  This 
is  done  by  a  "reverse  integration"  technique  (simple  for  this  exam¬ 
ple).  The  computed  risk  of  0.44  is  the  area  under  the  historical  pro¬ 
file  curve  from  the  required  productivity  point  over  all  intervals  to 
the  right  of  that  point.  For  the  current  example,  the  results  of 
this  process  are  illustrated  in  figure  6-9.  A  comparison  of  the 
acceptable  and  measured  risk  is  also  shown  in  figure  6-9. 

6.4.7  Tradeoff  Analysis  and  Reporting  Results:  Example. 

a.  The  final  step  in  the  RAMSS  process  is  to  look  for  possible 
consequences  of  the  acceptable  and  measured  risks.  If  the  conse¬ 
quences  are  not  acceptable,  then  alternatives  to  reduce  either  or 
both  risk  values  are  analyzed  for  cost,  schedule,  operational,  and 
support  impact.  Combinations  of  alternatives  (e.g.,  reducing  base¬ 
line  support  profile  requirements  and  improving  the  software  support 
factors)  is  a  likely  approach. 

b.  Consequences  of  the  acceptable  and  measured  risk  for  the  exam¬ 
ple  are  very  difficult  to  assess  without  more  details  concerning  the 
particular  decision  maker  focus  and  the  software  system's  operational 
use.  For  our  purposes  it  will  be  assumed  that  the  consequences  of 
this  risk  were  severe  enough  to  warrant  a  tradeoff  analysis  for  pos¬ 
sible  ways  to  reduce  the  risk.  Some  of  the  alternatives  which  could 
be  considered  for  the  example  are  illustrated  in  figure  6-10.  Part 
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of  the  reluctance  to  consider  a  particular  alternative  as  being 
effective  is  the  lack  of  specific  detailed  information.  Tradeoff 
analysis  is  not  a  cookbook  process.  The  preliminary  nature  of  the 
RAMSS  and  the  use  of  hypothetical  data  also  limits  confidence  in  the 
effectiveness  of  alternatives.  The  importance  of  the  example  is  to 
illustrate  that  logical  tradeoffs  can  be  considered,  and  the  impact 
upon  the  acceptable  and  measured  risk  can  be  graphically  illustrated 
in  a  reasonably  simple  way.  This  is  a  major  consideration  for  ade¬ 
quate  presentation  of  results  to  the  various  levels  of  decision 
makers. 

6.5  PROBLEMS  AND  OBSERVATIONS. 


The  framework  for  a  RAMSS  as  it  has  evolved  during  this  study 
and  as  it  is  reviewed  in  this  chapter  has  good  potential  for  practi¬ 
cal  application.  Some  problems  and  observations  are  listed  below: 

(1)  A  central,  valid  historical  maintenance  profile  data  base 
needs  to  be  developed.  This  data  collection  effort  sup¬ 
ports  the  capability  for  such  a  data  base  to  be  developed 
and  maintained  over  time. 

(2)  The  RAMSS  supportabi 1 ity  evaluation  upgrade  is  necessary. 
The  product  evaluation  would  be  least  affected.  The  life 
cycle  software  support  management  evaluation  was  consid¬ 
ered  to  be  important  by  personnel  surveyed. 

(3)  The  supportabi 1 ity  evaluation  metric  to  risk  conversion 
and  relationship  to  the  historical  baseline  maintenance 
profiles  needs  to  be  further  clarified. 

(4)  A  pilot  study  applying  the  RAMSS  framework  is  recom¬ 
mended. 
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SECTION  VII 

CONCLUSIONS/RECOMMENOATIONS 

7.1  INTRODUCTION. 

a.  In  addition  to  some  general  conclusions  and  recommendations 
from  this  study,  this  section  of  the  report  presents  some  important 
problems  which  were  observed  during  the  process  of  collecting 
software  support  activity  data  during  the  facility  visits. 

b.  All  of  these  problems  are  "negative,"  in  the  sense  that  their 
causes  interfere  with  or  detract  from  the  capabilities  of  the  various 
support  facilities  to  perform  their  missions.  This  in  no  way  implies 
that  the  support  facilities  are  not  performing  effectively.  But  they 
could  perform  more  effectively.  We  found  the  support  facilities 
staffed  by  qualified,  motivated  people  who  were  extremely  cooperative 
in  support  of  this  study-.  The  observations  and  problems  noted  below 
do  not  have  simple  solutions,  but  because  they  were  common  to  almost 
all  facilities  and  systems  examined,  it  seemed  appropriate  that  they 
be  documented.  An  improvement  in  some  of  these  areas  could  provide 
significant  enhanced  capability  for  software  support  at  some 
facilities.  Improvements  in  other  areas  would  make  the  data  collec¬ 
tion  for  future  studies  like  this  one  much  easier. 

c.  General  conclusions  and  recommendations  will  be  discussed 
concerning  the  data  collection  process,  the  software  supportabi 1 ity 
risk  computation,  historical  maintenance  profiles,  and  the 
supportabi 1 ity  problems  observed  during  the  site  visits.  Finally,  a 
list  of  conclusions  and  recommendations  from  this  study  will  be 
discussed. 
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7.2  DATA  COLLECTION  PROCESS. 

The  data  collected  during  this  study  effort  provided  valuable 
insight  into  the  software  support  process  at  various  sites.  There 
are  many  other  sites  which  were  not  visited,  and  a  large  number  of 
software  systems  for  which  data  were  not  collected.  In  order  for  the 
survey  data  to  remain  current  and  improve  in  quality,  it  is  necessary 
to  adapt  some  method  for  obtaining  new  data  and  updating  the  current 
historical  maintenance  profiles  and  the  associated  analysis  results. 
This  section  briefly  describes  the  conclusions  and  recommendations 
concerning  the  data  collection  process. 

7.2.1  Study  Effort  Oata  Collection.  The  data  collected  during  the 
study  effort  included  background,  evaluation,  and  software  release 
data  from  each  software  system  as  well  as  observation  and  interview 
data  from  the  site  visits.  •  The  conclusions  concerning  the  data 
collection  process  are  summarized  below. 

(1)  The  site  visit  was  essential  in  order  to  obtain  the  data. 

(2)  The  AFOTEC  preparation  prior  to  the  site  visit  was  very 
valuable  and  was  probably  the  main  reason  the  site 
personnel  were  so  cooperative  during  the  site  visit. 

(3)  Having  a  site  survey  form  was  essential  in  order  to 
quickly  focus  site  personnel  upon  the  purpose  of  the  sur¬ 
vey. 

(4)  The  background  and  evaluation  data  on  the  site  survey 
form  were  relatively  easy  to  collect,  with  the  exception 
of  an  estimate  of  supportabi 1 ity  risk;  the  problem  in 
this  case  was  lack  of  a  clear  definition  of  what  was 
being  assessed  for  risk.  As  the  site  visits  progressed, 
the  definition  of  this  risk  was  clarified  and  the  data 
became  more  consistent  and  accurate. 
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(5)  The  software  release  data  was  difficult  to  obtain.  This 
difficulty  was  primarily  due  to  the  necessity  to 
reconstruct  '~rior  release  data  rather  than  the  inability 
to  record  such  data.  It  was  determined  that  the  release 
data  requested  could  be  easily  recorded  during  the 
release  effort. 

(6)  In  general,  the  data  obtained  are  probably  of  somewhat 
low  validity.  That  is,  the  consistency  of  the  data  is 
weak  and  tne  accuracy  of  the  data  is  low.  Variance 
within  the  data  is  too  high  for  statistically  strong 
significant  results.  However,  it  appeared  that  most  of 
the  more  important  data  could  be  improved  a  great  amount 
through  a  more  regular  data  collection  effort  (during  the 
release  effort) . 

(7)  Storage  of  the  collected  data  in  a  dBASE  III  data  base 
was  very  valuable  and  allowed  easy  generation  of  reports 
for  analysis  review. 

7.2.2  Recommended  Future  Data  Collection  Procedure.  It  is  necessary 
to  continue  to  collect  data  similar  to  the  site  survey  data.  It  is 
also  necessary  to  make  the  data  collection  process  efficient  for  site 
personnel  and  somewhat  related  to  activity  already  being 

accomplished.  A  recommended  data  collection  form  and  procedure  is 
discussed  in  section  III.  The  essential  elements  of  the  form  and 
procedure  are: 

(1)  The  form  and  procedure  are  temporary  until  a  mere 

permanent  arrangement  can  be  integrated  into  the  Air 
Force  software  support  concept. 

(2)  All  cognizant  software  support  sites  and  major  (critical) 
software  systems  currently  being  supported  should  be 
required  to  participate  in  the  data  collection  effort. 
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(3)  It  is  estimated  that  completion  of  the  data  collection 
form  (and  altering  current  practices  so  the  data  are 
readily  available)  would  take  very  little  additional 
personnel  time.  The  range  might  be  from  one  person  day 
to  one  person  week  per  release. 

(4)  The  data  collection  form  data  elements  required  for  each 

release  include:  site,  system,  software  system,  software 
system  type,  size  in  thousands  of  source  lines,  source 
languages,  personnel  counts  and  skill  levels,  release 
identifications,  release  start  dates,  engineering 

completion  dates,  field  release  dates,  and  baseline 
software  support  profile  data  on  each  change  request  in 
the  release. 

(5)  The  data  collection  procedure  would  involve  each  support 
site  completing  a  data  collection  form  for  each  software 
system  release.  The  form  would  be  sent  to  a  data 
repository  site  (AFOTEC,  at  this  time)  for  integration 
into  the  current  data  base,  update  of  the  historical 
maintenance  profiles,  and  further  statistical  analysis. 

(6)  It  is  recommended  that  such  a  data  collection  form  be 
adapted  and  that  AFOTEC  develop  the  necessary  data  base 
and  analysis  environment  to  support  regular  revisions  to 
the  historical  maintenance  profiles. 

7.3  SOFTWARE  SUPPORTABILITY  RISK  COMPUTATION. 

a.  The  initial  approach  of  the  RAMSS  to  software  supportaoi 1 i ty 
risk  computation  is  reported  in  reference  8.3.  This  approach  used  a 
simple  linear  conversion  of  the  software  supportabi 1 ity  evaluation 
metrics.  Analysis  was  conducted  to  determine  the  validity  of  tnis 
approach,  and  perhaps  derive  alternative  approaches.  Details  of  this 
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analysis  are  presented  in  section  IV.  Essential  conclusions  and 

recommendations  from  this  analysis  are  discussed  below. 

(1)  The  analysis  indicated  very  little  support  for  using  a 

linear  conversion  of  the  software  supportabi 1 ity  metrics. 
The  boundary  cases  were  supported,  but  the  data  were 

better  fit  by  a  curved  line  than  a  straight  one.  Details 
of  this  analysis  are  presented  in  section  4.4.1. 

(2)  A  linear  regression  model  was  derived  which  more 

accurately  described  the  relationship  between  the  soft¬ 
ware  supportabi 1 ity  metric  (overall  general  score)  and 

software  supportabi 1 ity  risk.  The  regression  equation  is 
described  in  section  4. 4. 1.2  and  can  be  reasonably  easily 
used  for  computation  of  risk  with  a  hand  calculator. 

(3)  A  factor  regression  model  was  derived  which  even  more 

accurately  described  the  relationship  between  the 
software  supportabi 1 ity  metrics  and  software  support- 
ability  risk.  The  factor  regression  model  was  based  on 

six  derived  factors,  and  four  factors  (support 

management,  product,  personnel,  and  software  support 
systems)  of  the  six  were  significant  drivers  of  software 
support  risk.  The  factor  regression  model  and  equation 
are  described  in  sections  4.4.2  and  4.4.3.  The 

computation  of  software  supportabil ity  risk  is  relatively 
involved,  but  is  easily  accomplished  using  a  computer. 

b.  A  comparison  of  the  three  models  for  computing  software  sup- 

portability  risk  is  presented  in  section  4.4.4.  The  recommended 

model  is  the  factor  analysis  regression  model.  This  model  has  the 

2 

highest  R  value  which  accounts  for  the  variation  of  the  transformed 

9 

risk  values  about  their  mean.  It  should  be  cautioned  that  the  R“  is 
not  very  "strong"  for  any  of  the  models  and  care  should  be  taken  in 
using  them. 
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Ic.  The  linear  regression  model  and  the  simple  linear  model  can  be 
used  as  a  check  against  the  factor  regression  model  as  future  data 
are  collected  and  the  analysis  results  updated.  Because  of  the 
comparatively  low  confidence  in  the  data  itself,  the  updated  analysis 
may  show  more  complete  and  surely  more  valid  results. 


7.4  MAINTENANCE  PROFILES  AND  RELEASE  DATA. 

a.  The  historical  maintenance  profiles  are  derived  from  the  soft¬ 
ware  system  release  data.  These  profiles  are  a  histogram  with  an 
X-axis  representing  discrete  ranges  of  person-months  per  change  and 
the  Y-axis  representing  a  count  of  the  number  of  software  system 
releases  in  each  of  the  discrete  X-axis  ranges.  The  historical 
maintenance  profiles  are  presented  in  section  V  and  the  analysis 
results  of  the  software  release  data  are  presented  in  section  4.5. 
Essential  conclusions  and  recommendations  from  this  analysis  are  dis¬ 
cussed  below. 


b.  The  historical  maintenance  profiles  were  constructed  for  all 
software  system  releases,  and  also  for  stratified  subsets  by  site  and 
by  software  system  type.  Regression  analysis  was  then  performed 
using  person-months  per  change  as  the  dependent  variable  and  various 
independent  variables.  The  analysis  attempted  to  determine  if  a 
linear  regression  model  existed  for  predicting  person-months  per 
change  from  the  independent  variables  rather  than  computing  the  value 
as  a  ratio  (whose  terms  exist  only  after  the  release  is  done,  not 
prior  to  the  release).  Independent  variables  included  number  of 
source  lines  in  thousands,  proportion  of  correction  change  requests, 
proportion  of  changes  of  low  complexity,  proportion  of  c.nanges  of 
normal  priority,  percentage  of  high-level  language  source  code 
average  skill  level  of  personnel,  and  indicator  variables  for 
software  types. 
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c.  Results  from  the  regression  analysis  showed  no  clear, 
consistent  relationship  between  available  person-months  per  change 
and  the  independent  variables,  save  for  those  indicating  the  several 
software  types.  Because  software  type  was  shown  to  be  a  significant 
•actor  with  -aspect  to  avai1ab'e  person-months  oe-  change,  separate 
profiles  should  be  used  for  the  various  software  types  in  assessing 
risk.  Although  none  of  the  other  independent  variables  were  shown  to 
-elate  to  available  oerson-months  per  change,  tne  possibility  still 
exists  that  given  better  data,  significant  relationships  mignt  oe 
found.  Thus  it  is  recommenced  that  these  analysis  techniques  be  used 
during  the  future  update  of  the  data  on  software  releases  to  confirm 
or  negate  the  hypotheses  which  are  briefly  outlined  below.  Use  of 
actual  person  montns  oer  change  (as  specified  in  the  recommended  data 
collection  -'arm)  as  oooosed  to  avai ' ab'e  oerson-months  per  change  may 
also  affect  the  analysis  results. 

(1)  Hypothesis  l.  The  percentage  of  low  complexity  change 

requests  in  a  software  release  is  negatively  correlated 
with  the  person-months  per  change.  That  is  the  higner 
percentage  of  low  complexity  change  requests,  the  lowe- 
the  person-months  per  change. 

(2)  Hypothesis  2.  The  percentage  of  hign-level  language 
source  lines  in  a  software  release  is  [weakly;  negatively 
correlated  with  the  cerson-mcnths  oer  change  request. 

(3)  Hypothesis  3.  The  average  skill  level  of  personnel,  oro- 
uorfion  of  correction  change  requests,  and  proportion  of 
normal  priority  cnange  requests  are  not  correlated  with 
oe-son-montn$  oe-  change. 

d.  The  first  two  hypotheses  can  be  explained  in  a  rathe- 
straightforward  manner  since  complexity  is  directly  tied  to  e-fort 
level  and  it  is  generally  accepted  that  it  is  easier  to  maintain 
software  written  in  a  high  orde-  language. 
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e.  The  third  hypothesis  is  not  as  easily  explained,  but  a 
specific  understanding  of  the  dependent  variables  helps  build  a 
logical  rationale.  There  is  no  doubt  that  normal  priority  changes 
.received  lower  priority,  but  there  was  definitely  no  trend  that  indi¬ 
cated  normal  oriority  reauests  were  any  more  or  less  difficult  to 
accomplish  than  non-normal  (urgent  and  emergency)  change  requests.  A 
similar  statement  holds  for  correction  change  requests.  However,  it 
would  appear  that  the  greater  the  average  skill  level,  the  more 
quickly  change  requests  could  be  completed.  In  a  more  pure  environ¬ 
ment  that  may  be  true,  but  it  is  a  rare  environment  in  which  person¬ 
nel  skills  do  not  range  over  the  full  scale  of  1  (lowest)  to  5 
(highest).  Furthermore,  the  more  highly  skilled  personnel  are  given 
the  more  difficult  changes,  which  tends  to  result  in  similar  person- 
months  per  change  values.  Finally,  most  changes  in  a  release  are  of 
low  complexity  and  many  times  it  takes  more  highly  skilled  personnel 
about  the  same  amount  of  time  to  complete  such  changes  as  it  does  the 
average  personnel . 

f.  In  argument  against  the  average  skill  level  of  personnel  hypo¬ 
thesis,  it  is  strongly  suspected  that  the  average  skill  level  -  and 
percentage  of  low  complexity  change  requests  may  in  combination 
relate  to  person-months  per  change.  This  is  one  reason  it  is 
recommended  that  skill  level  of  personnel  be  collected  during  future 
data  collection  efforts. 

g.  It  is  recommended  that  the  historical  maintenance  profiles  be 
continually  updated  with  data  derived  from  the  recommended  future 
data  collection  procedure  and  associated  form.  All  support  sites  and 
major  software  systems  snould  be  represented  in  future  historical 
maintenance  profiles. 
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7.5  GENERAL  PROBLEMS  OBSERVED  DURING  SITE  VISITS. 

a.  The  problems  discussed  below  were  noted  at  most  facilities, 
and  hence  have  general  application.  When  possible,  specific  recom¬ 
mendations  for  improvement  are  noted. 

b.  The  problems  discussed  in  this  section  are  summarized  below: 

Table  7-1. 

Summary  of  Problems 


Problem 

1.  High  personnel  attrition  rate 

2.  Inconsistent  application  of  software 
configuration  management  system  (CMS) 

3.  Lack  of  agreement  on  definition  of 
terms 

4.  Modification  to  software  takes  too 
long  to  reach  the  field 


5.  Systems  support  dedicated  to 
multiple  ALCs 

6.  Contractors  not  transferring 
responsibility  for  software 
support 

7.  Constraints  on  OFP  software 
development 


Solution 

Requires  further  study 

Set  and  enforce  standard 
CMS  procedure 

Terms  defined  as  part  of 
standard  CMS 

Study  modification  and 
testing  process  to 
streamline 

Requires  concept  change 
for  ALC  operation 

Need  more  information 


None.  Problem  recognition 
required 


7.5.1  Personnel  Attrition  Rate. 

Problem 

a.  One  of  the  most  common  problems  noted  was  the  high  attrition 
rate  of  personnel  assigned  to  software  support.  The  biggest  contri- 
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butor  to  this  problem  appeared  to  be  the  higher  salaries  and  career 
opportunities  available  from  government  contractor  organizations. 
Some  support  organizations  were  severely  affected  by  this  problem. 
Turnover  was  reported  as  being  between  20  percent  and  40  percent  per 
year  in  some  instances.  To  a  lesser  degree,  the  duty  rotation  of 
military  personnel  also  had  significant  impact.  Due  to  the  highly 
technical,  complex  nature  of  emerging  weapons  systems,  the  training 
required  to  bring  new  personnel  to  an  effective  level  of  job 
performance  is  very  time  consuming.  When  it  takes  13  months  to  train 
a  new  employee  and  he  or  she  leaves  after  3  years,  only  to  be 
replaced  by  a  new  employee  who  needs  training,  the  overhead  is  high. 
Many  young  military  personnel,  if  they  are  not  lost  due  to  transfer, 
are  lost  by  obtaining  higher  paying  jobs  with  government  contractors 
at  the  completion  of  their  military  obligation. 

Solution 


b.  There  is  no  simple  solution  to  this  problem.  It  would  seem 
that  salaries  and  career  opportunities  for  government  employees  are 
not  as  great  as  for  civilian  contractors,  therefore  improvement  in 
pay  and  benefits  are  significant  issues.  On  the  other  hand,  one 
could  argue  that  civilian  contractors  are  overpaid,  and  that  action 
should  be  taken  to  remedy  this  situation.  It  is  not  in  the  scope  of 
this  study  to  address  this  problem,  except  to  mention  that  the 
problem  exists,  and  that  it  is  serious.  Some  relief  for  military 
personnel  might  be  to  lengthen  the  active  duty  tour  at  software 
support  facilities.  However  this  solution  has  potential  negative 
effects  on  military  careers.  An  .independent  study  should  be  made 
concerning  the  attrition  issue,  and  specific  recommendations  made  for 
its  resolution. 
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7.5.2  Inconsistent  Application  of  Software  Configuration  Management- 


Problem 

a.  There  are  really  two  problems  here.  The  first  problem  is  that 
no  software  support  facility  that  was  visited  was  tracking  or 
recording  software  maintenance  data  in  a  manner  consistent  with  that 
of  any  other  facility.  Each  software  system  was  using  its  own  method 
for  performing  software  configuration  management.  One  bright  spot  in 
the  study  was  that  at  least  everyone  seemed  to  be  using  some  form  of 
software  configuration  management.  However,  it  was  expected  that  at 
least  the  configuration  management  systems  used  on  different  software 
systems  located  at  the  same  facility  in  the  same  building  managed  by 
the  same  organization  would  have  high  degrees  of  similarity.  This 
was  simply  not  true.  There  appears  to  be  a  great  amount  of  disagree¬ 
ment  surrounding  what  items  should  be  monitored  and  what  information 
should  be  tracked  by  a  configuration  management  system. 

b.  The  second  problem  with  the  maintenance  data  was  caused  by  the 
first.  It  was  extremely  difficult  to  collect  the  data  needed  for 
this  study,  because  such  data  was  very  rarely  maintained  in  a  form 
that  was  readily  accessible.  The  desired  information  was  usually 
present,  but  the  inconsistency  in  storage  methods  necessitated  extra 
time  to  extract  it.  Some  configuration  methods  were  mostly  auto¬ 
mated,  some  partially  automated,  and  some  almost  totally  manual.  The 
automated  methods,  when  done  properly,  were  the  easiest  and  quickest 
methods  for  obtaining  desired  information.  Had  there  been  at  least  a 
consistent  configuration  management  system,  the  data  would  have  been 
easier  to  extract.  This  condition,  unless  rectified,  will  continue 
to  be  a  problem  for  efforts  by  AFOTEC  to  establish  new  historical 
profiles  in  future  years. 
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Solution 


c.  These  problems  are  workable.  It  is  not  expected  that  each 
software  configuration  management  system  (CMS)  collect,  store,  and 
retrieve  the  same  information  using  the  identical  methodology.  What 
is  reasonable,  however,  is  that  each  CMS  maintain  some  finite  set  of 
information  consistently  and  accurately  across  all  software  systems. 
There  is  a  lot  of  literature  currently  available  about  CMS.  What  is 
lacking  is  a  standard  set  of  procedures  to  be  used  and  data  items  to 
be  collected.  If  this  standard  were  set  and  enforced  for  all  govern¬ 
ment  software  support  facilities,  then  future  studies  such  as  this 
one  would  be  much  easier.  However,  the  reason  for  setting  a  standard 
should  not  be  to  accommodate  studies.  The  reason  should  be  that  a 
good  CMS  means  software  maintenance  is  under  control  and  is  being 
properly  performed.  A  good  CMS  provides  (among  other  things)  the 
maintenance  effort  with  a  management  tool  that  increases  the  ability 
to  properly  allocate  limited  resources  while  decreasing  the 
probability  that  errors  are  introduced  during  the  maintenance 
process.  A  separate  study  should  be  performed  which  recommends  a 
standard  set  of  CMS  procedures  and  data  items  for  DoD  software 
support  facilities,  with  the  data  desired  by  this  study  as  a 
preferable  start  point.  Given  a  standard  CMS,  AFOTEC  could  collect 
the  information  to  update  future  baselines  without  having  to  make 
costly  facility  visits.  An  initial  approach  to  solving  this  problem 
is  discussed  in  section  3.6  of  this  report. 

7.5.3  Lack  of  Definition  Agreement. 

Problem 


a.  This  problem  may  oe  a  contributory  factor  to  tne  problems 
discussed  in  section  7.5.2,  in  the  sense  that  there  has  been 
disagreement  in  the  software  community  about  what  activities  should 
be  tracked  or  called  software  maintenance.  This  study  recommends 


*1 


BDM/A-35-0510-TR 


THE  BDM  CORPORATION 


that  the  categories  of  maintenance  requests  be  priority  type,  main¬ 
tenance  type,  and  complexity  level.  This  study  also  recommends  that 
the  software  maintenance  types  be  called  corrections,  enhancements, 
or  conversions.  Some  authorities  in  the  software  community  do  not 
consider  enhancements  to  be  software  maintenance  activities  at  all. 
In  addition,  this  study  showed  that  many  of  the  software  support 
maintenance  activities  which  should  be  categorized  as  conversions 
were  in  fact  called  corrections  or  enhancements  by  the  support 
facilities.  In  the  same  manner,  the  categorization  of  priority  into 
emergency,  urgent,  or  normal,  and  that  of  complexity  level  into  high, 
medium,  or  low,  are  not  clearly  understood.  Hence,  the  lack  of  a 
standardized  set  of  definitions  is  inhibiting  the  collection  of 
common  data  across  maintenance  systems. 

Solution 

b.  This  study  proposes  definitions  .  for  the  above  categories  of 
software  maintenance  activities.  The  glossary  (appendix  B)  contains 
definitions  of  the  controversial  terms.  These  definitions  are 
summarized  in  table  7-2.  While  these  definitions  may  be  argued,  they 
have  at  least  been  applied  consistently  in  this  study  and  are  recom¬ 
mended  for  adoption. 

7.5.4  Software  Release  Process  Takes  Too  Long.' 

Problem 

a..  There  were  several  instances  during  this  study  when  mention 
was  made  of  the  unreasonably  long  time  that  is  required  for  a  soft¬ 
ware  modification  to  be  completed  and  put  into  the  field.  This  time 
is  measured  from  the  date  the  software  support  facilities  actually 
begin  work  on  the  modification  to  the  time  it  is  operationally 
available.  Current  figures  indicate  that  18  months  is  about  the 
average  time  for  this  to  occur,  of  which  about  11  months  is  software 
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Table  7-2. 

Definition  of  Maintenance  Activity  Terms 

Maintenance  Request  Category  --  the  identification  of  a  maintenance 
request  by  specification  of  the  priority  type,  maintenance  type, 
and  complexity  level . 

Maintenance  Type  --  the  type  of  maintenance  actions  required  to 
complete  a  maintenance  request:  enhancement,  conversion, 

correction . 

Corrective  Maintenance  Action  (MA)  --  any  change  which  is  neces¬ 

sitated  by  actual  faults  (induced  or  residual)  in  a  software 
system. 

Enhancement  (perfective)  MA  --  any  change,  insertion,  deletion, 
modification,  extension,  and  enhancement  made  to  a  software  system 
to  meet  the  evolving  needs  of  the  user. 

Conversion  (Adaptive)  MA  --  any  change/effort  to  a  software  system 
which  is  initiated  as  a  result  of  changes  in  the  environment 

(e.g.,  hardware,  system  software)  in  which  the  software  system 
must  operate. 

Priority  Type  --  the  criticality  of  the  maintenance  request  in  order 
to  preserve  mission  readiness:  emergency,  urgent,  normal. 

Emergency  MA  —  an  MA  requiring  all  available  personnel's  dedicated 

effort,  to  correct  the  problem  as  soon  as  possible  (e.g., 

24  hours);  MIL-STD-1679  severity  code  1  or  2:  mission  termination 
or  severe  degradation 

Urgent  MA  --  an  MA  requiring  next  "block  release"  turnaround; 
MIL-STD-1679  severity  code  3:  mission  impact 

Normal  MA  --  an  MA  not  in  the  Emergency  or  Urgent  categories; 
MIL-STD-1679  severity  code  4  or  5:  mission  inconvenience 

High  Complexity  MA  --  an  MA  where  changes  are  in  requirements , 

design,  code,  and  test;  or  >  10  percent  of  CSCI  is  affected;  or 
several  modules  are  affected  by  the  change  (global  changes);  or 
the  technical  nature  of  the  change  requires  highly  specialized 
personnel  skills;  or  the  level  of  effort  by  personnel  is  large 

Medium  Complexity  MA  --  an  MA  where  changes  are  in  design,  code  and 
test;  or  between  1  percent  and  10  percent  of  CSCI  is  affected;  or 
at  least  two  modules  are  affected  by  the  change  (semi  -  local ) ;  or 
the  level  of  effort  by  personnel  is  average. 

Low  Complexity  MA  --  an  MA  where  changes  are  isolated  to  only  one 
unit  (e.g.,  one  module/compilation  unit)  or  code;  or  no  more  than 
1  percent  of  CSCI  is  affected;  or  the  level  of  effort  by  personnel 
is  minimal 
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modification  and  7  months  is  field  (operational)  testing.  Many  felt 
that  this  period  of  time  could  be  decreased,  with  considerable 
benefits  to  the  weapon  system. 

Solution 


b.  This  problem  has  no  easy  solution.  Most  personnel  interviewed 
were  frustrated  by  the  (to  them)  unreasonable  time  to  perform 
operational  testing.  However,  there  are  undoubtedly  many  constraints 
in  this  area  that  support  facilities  are  simply  not  aware  of.  It  is 
recommended  that  this  problem  be  given  further  study. 

7.5.5  Multiple  ALCs  Supporting  Same  System. 


Problem 


a.  Several  instances  were  reported  in  which  one  ALC  may  be  the 
OPR  for  a  given  software  system,  but  another  ALC  or  organization  may 
be  doing  the  support  function.  At  other  times,  changes  to  software 
being  supported  at  one  ALC  may  affect  another  system,  and  hence  the 
software  support  of  that  system,  at  another  ALC  or  organization. 
This  situation  often  creates  tremendous  coordination  and  communi¬ 
cations  problems,  further  complicating  the  already  difficult  job  of 
adequately  supporting  software  changes. 

Solution 


b.  Due  to  the  construct  of  the  ALC  system,  where  weapons  systems 
are  spread  around  to  the  various  ALCs,  this  problem  is  unlikely  to  be 
reduced  unless  there  is  a  change  in  support  location  philosophy. 
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7.5.6  Contractor  .Not  Transferring  Responsibility  for  the  Software. 
Problem 


a.  In  some  cases,  the  support  facilities  have  been  staffing  and 
training  personnel  in  anticipation  of  performing  the  software  support 
function,  only  to  have  the  PMRT  date  slipped  for  one  reason  or 
another.  Sometimes  the  contractor  is  still  performing  maintenance 
under  a  "guarantee"  agreement.  For  the  F - 16  WST  at  Ogden  ALC,  the 
software  support  team  has  been  providing  interim  releases  of  software 
change  requests  which  the  contractor  cannot  get  into  the  baseline 
releases.  Under  continually  slipping  PMRT  dates,  the  government 
support  facilities  have  found  it  difficult  to  establish  the  baselines 
for  software  support  required  to  perform  the  maintenance  function. 

Solution 


b.  Without  knowing  more  about  why  the  systems  are  not  undergoing 
PMRT,  it  is  difficult  to  recommend  an  overall  fix  for  this  problem. 
However  the  problem  was  mentioned  often  enough  to  warrant 
recognition. 

7.5.7  Constraints  on  OFP  Software  Development. 

Problem 


a.  Many  times  the  problem  with  making  software  changes  to  F-4  and 
F- 16  operational  systems  (as  well  as  other  OFP  systems)  is  as 
dependent  upon  the  memory  size  available  for  the  systems  as  upon  the 
availability  of  the  people  to  do  the  work.  Often,  adding  enhance¬ 
ments  may  require  the  unplanned  complete  removal  of  an  existing 
capability  just  to  create  the  memory  space  to  make  the  enhancements. 
This  restriction  creates  unique  problems  in  embedded  software. 
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Solution 


b.  There  is  no  immediate  solution  to  this  problem.  As  hardware 
technology  improves  to  allow  more  memory  space  for  software,  some  of 
this  problem  may  be  resolved.  For  the  time  being  all  one  can  do  is 
recognize  the  problem  and  make  the  best  of  it. 

7.5.8  General  Observations. 

a.  The  following  comments  cannot  really  be  categorized  as 
problems,  but  are  instead  observations  about  the  apparent  character¬ 
istics  of  software  support  activities. 

b.  At  the  beginning  of  this  study,  one  of  the  elements  of  the 
desired  software  maintenance  activity  data  to  be  collected  was  the 
actual  person-hours  per  change.  After  several  attempts  to  collect 
this  data,  it  became  clear  that  the  data  was  not  available.  Two 
basic  reasons  for  this  were:  1)  There  is  no  good  mechanism  for 
recording  such  information,  and  2)  when  people  work  on  software 
projects  (or  almost  any  project  for  that  matter)  it  is  very  difficult 
to  define  which  time  should  be  counted  toward  completing  the  task. 
It  was  learned  during  the  facility  visits  that  many  activities  are 
being  performed  by  software  maintenance  personnel  which  are  not 
directly  software  maintenance  activities.  For  accountability  of  time 
and  resources,  it  was  apparent  from  the  maintenance  data  that  actual 
logged  project  maintenance  time  and  actual  project  configuration 
management  time  are  a  small  part  of  the  overall  support  time.  Such 
things  as  waiting  for  testing  on  a  simulator,  waiting  for  a  simulator 
to  be  repaired,  lack  of  work  because  the  facility  is  staffed  but  the 
system  has  not  been  released  to  the  government,  or  training  time  (new 
and  old  personnel),  etc.,  are  not  directly  acountable.  Other  items 
which  are  not  logged  include  review  and  support  of  externally 
requested  tests,  monitoring  contractor  development/delivery/support, 
political  strategy  meetings,  and  update  of  documentation  and 
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facilities.  The  amount  of  time  spent  actually  performing  software 

change  activity  may  vary  from  30  percent  to  90  percent  of  the  time 
available. 

c.  Due  to  the  unavailability  of  person-hours  per  software  change, 
the  unit  of  measure  for  productivity  became  available  person-months 
per  block  release.  Each  block  release  is  composed  of  a  group  of 
changes.  This  measure  is  computed  by  knowing  how  long  the  block 

release  was  worked,  and  how  many  personnel  were  assigned  to  the  task 

for  that  length  of  time.  Then  the  baseline  ("cost"  in  person-months 
per  change  as  a  function  of  number  of  block  releases  with  that 

"cost")  includes  all  of  the  overhead  time  mentioned  in  the  previous 
paragraph.  This  measure  is  more  reflective  of  the  time  that  is 
actually  spent  in  doing  the  total  software  support  function  and,  in 
the  final  analysis,  may  be  more  realistic. 

7.6  CONCLUSIONS/RECOMMENDATIONS. 

a.  It  should  be  clear  that  the  process  of  providing  software 
support  for  major  weapons  systems  is  a  complex  and  highly  labor- 
intensive  task,  with  a  lot  of  room  for  improved  methods  of  providing 
high  quality,  cost  effective  systems  on  a  timely  basis.  This  report 
has  discussed  some  issues  which  were  not  directly  related  to  how  to 
perform  risk  assessment.  But,  the  resolution  of  these  issues  would 
reduce  the  risk  of  software  supportabi 1 ity  should  a  risk  assessment 
be  performed.  These  issues  include  high  personnel  attrition  rate, 
ineffective  configuration  management  systems,  inconsistent  termi¬ 
nology  applications,  lengthy  time  to  produce  operational  changes, 
multiple  Al_C  support,  delay  of  PMRT,  and  poor  constraints  on  OFP 
software  development. 


b.  The  most  significant  conclusion,  which  supports  the  primary 
goal  of  this  report,  is  that  the  results  of  this  study  indicate 
that  the  software  support  data  justify  further  refinement  and 
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verification  of  a  risk  assessment  model.  Analysis  of  the  data 
indicates  that,  by  virtue  of  this  study,  the  methodology  has  been 
improved  and  is  now  ready  for  partial  application  via  a  pilot  study. 
Full  application,  is  not  possible  because  the  software  life  cycle 
evaluation  tool  has  not  been  developed,  and  the  other  evaluation 
tools  require  update  to  conform  to  the  RAMSS.  The  following  ordered 
steps  are  required  to  provide  a  RAMSS  which  fulfills  the  total 
objectives: 

(1)  Adapt  the  software  maintainability  tool  to  measure 
against  the  user/supporter  baseline. 

(2)  Adapt  the  software  support  facility  evaluation  tool 
(ASSET)  to  measure  against  the  user/supporter  baseline 
agreement. 

(3)  Complete  the  top  level  software  life  cycle  process 
evaluation  metrics  for  risk  assessment  (RA). 


(.4)  Apply  the  software  supportabi  1  ity  risk  assessment  method¬ 
ology  to  a  current  program.  Evaluate  results  of  the 
application  and  complete  technology  transfer  to  AFOTEC 
personnel . 

(5)  Continue  the  collection  of  software  system  release  data. 

(6)  Develop  procedures  to  update  the  historical  maintenance 
profiles  and  analysis  results  from  the  newly  collected 
software  system  release  data. 

(7)  Continue  to  evolve  the  software  supportabi 1 i ty  risk 
computation  regression  analysis  results.  Use  the  factor 
regression  model,  the  linear  regression  model,  and  the 
simple  linear  model  in  that  order  of  preference. 


VI 1-19 


1  -'•j.'-  .v .V- /■ 


•?••>»»» 


THE  BOM  CORPORATION 


BDM/ A-35-Q51Q-TR 


(8)  Continue  to  analyze  potential  relationships  between 
person-months  per  change  and  other  system- level 
variables,  such  as  percentage  of  low  complexity  change 
requests . 
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ACM 

ADPF 

ADTS 

AF 

AF5 

AFOTEC 

AGE  OP 

AIS 

AISF 

ALC 

ALCM 

AOCP 

APT 

AS  IT 

ASSET 

ATC 

ATO 

ATE 

AWACS 

3  MOP 

BNST 

3TG 

C-E 

C3I 

CAOC 

CAFMS 

cc 

COR 

Cl 

CITS 


APPENOIX  A 
LIST  OF  ACRONYMS 

Air  Combat  Maneuvering 

Automatic  Data  Processing  Facility 

Avionic  Oepot  Test  Station 

Air  Force 

Air  Force  Base 

Air  Force  Operational  Test  and  Evaluation  Center 

Aerospace  Ground  Equipment  Operating  System 

Avionics  Intermediate  Shop 

Avionics  Integration  Support  Facility 

Air  Logistics  Center 

Air  Launched  Cruise  Missile 

Airborne  Operational  Computer  Program 

Available  Person  Time 

Adaptable  Surface  Interface  Terminal 

AFOTEC  Software  Support  Evaluation  Tool 

Air  Training  Command 

Aircrew  Training  Device 

Automatic  Test  Equipment 

Airborne  Warning  and  Control  System 

3MDP  Statistical  Software  (NOTE:  BMDP  is  a  name,  not 

acronym. ) 

Bomb  Navigation  Station  Trainer 

Blue  Tape  Generator 

Ccmmun icat i ons-E  lectron ics 

Command /Control /Communications /Intel  1 igence 

Central  Air  Data  Comouter 

Computer  Assisted  Force  Management  System 

Central  Computer 

Critical  Design  Review 

Configuration  Item 

Central  Integrated  Test  System 
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CM 

CMP 

CMS 

CPT 

CRISP 

CSCI 

css 

DBR 

DC/SR 

OPS 

DT&E 

OOD 

ECS 

EMUX 

EVS 

EW 

F/CGMS 

FCC 

FCR 

FESP 

FTSS 

FTSS 

GLCM 

HQ  TAC 

HUD 

II 

INS 

IOC 

ISA 

I  vu 
JT  IDS 
L/P 
LIT 
LASER 


Conf iguration  Management 

Configuration  Management  Plan 

Configuration  Management  System 

Cockpit  Procedure  Trainer 

Computer  Resources  Integrated  Support  Plan 

Computer  Software  Conf igurat ion  Item 

Communications  System  Segment 

Engineering  Duration  of  the  Block  Release  (months) 

Display  Control/Storage  Retrieval 

Data  Processing  System 

Development  Test  and  Evaluation 

Department  of  Defense 

Embedded  Comouter  System 

Electrical  Multiolex  System 

Electro-Optical  Visual  System 

Electronic  Warfare 

Fuel/Center  of  Gravity  Management  System 
Fire  Control  Computer 
Fire  Control  Radar 

Facility,  Equipment,  and  Software  Support  Plan 

Flight  Test  Simulation  System 

Flight  Test  Support  System 

Ground  Launched  Cruise  Missile 

Headquarters  Tactical  Air  Command 

Head-Up  Display 

Imagery  Interpretati on 

Inertial  Navigation  System 

Intial  Operational  Capability 

Interface  Simulator  Analyzer 

Independent  Verification  and  Validation 

Joint  Tactical  Information  Distribution  System 

Line  Printer 

Level  1  Test 

Light  AmDlication  by  Stimulated  Emission  of  Radiation 
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M-OTO 


MARRES 
MC-1  EXEC 
MC-2  EXEC 


NORAD 


OC-ALC 


OO-AIC 


O/SCMP 


PAVE  TACK 


PMPCHNG 


Loaded  Pylon  Test 
Line  Replaceable  Unit 
Maintenance-Oata  Transport  Oevice 
Maintenance  Action 

Manual  Radar  Reconnaissance  Exploitation  System 

B- 52  Block  I  OFP  Exec 

8-52  Block  II  OFP  Exec 

Modular  Display  Sub-System 

Mission  Data  Transfer  System 

Mission  Essential  Backup 

Missile  Procedures  Trainer 

NORAD  Computer  System 

North  American  Aerospace  Defense  Command 

Number  of  Persons  Assigned  to  Software  System 

Oklahoma  City  ALC 

Operational  Computer  Program 

Operational  Flight  Program 

OpeationaT  FI ightcrew  Trainer 

Ogden  ALC 

Office  of  Primary  Responsibility 
Offensive  Radar  System 

Operational/Support  Configuration  Management  Plan 
Off-Site  Test  Facility 
Operational  Test  and  Evaluation 
Electronic  Optical  System  for  Laser  Guided  Weapons 
Percentage  of  the  Persons  Dedicated  to  the  Block  Release 
'  Percentage  of  the  Persons  Dedicated  to  the  Software  System 
Program  Management  Directive 
Program  Management  J!an 
Person-Months  per  Change 
Person-Months  pe^  Change 
Program  Management  Responsibility  Transfer 
Program  Office 

Programmable  Read-Only  Memory 
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RAMSS 


SM-ALC 


SP/USER 
SPACE COM 


SRGSCP 


STRTS 


TEREC 


TPOCP 


WR-ALC 


Quality  Assurance 
Risk  Assessment 

Risk  Assessment  Methodology  for  Software  Supportabi 1 ity 

Radio  Frequency 

Software 

Simulation 

Sacramento  ALC 

System  Maintenance  Computer  Program 
Stores  Management  System 
Signal  Processor  User  Simulator- 
Space  command 

Short  Range  Attack  Missile 

Surveillance  Radar  Computer  Program 

Surveillance  Radar  Ground  Support  computer  Progream 

Software  Supportabi 1 ity 

Software  Support 

Space  Surveillance  Center 

Software  Support  Facility 

Simulator  Tactical  Radar  Training  System 

Support  Utility 

System 

Tactical  Air  Control  System 

Time  to  Complete  Maintenance  Request 

Test  and  Evaluation  Master  Plan 

Tactical  Electronic  Reconnaissance 

Tactical  Information  Processing  Interpretation 

Translator  Process  Operational  ComDuter  Program 

Utility 

Unit  Under  Test 
Weapon  Control  System 
Weapon  Navigation  Computer 
Warner  Robins  ALC 
Weapon  System  Trainer 
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APPENDIX  B 
GLOSSARY  OF  TERMS 


B. 1  INTRODUCTION. 

a.  The  glossary  of  terms  for  the  RAMSS  has  varied  as  the 

methodology  development  has  progressed.  Refer  to  BDM/A-84-322-TR 
(Final)  dated  September  28,  1984,  for  a  complete  glossary  of  terms 
relating  to  risk  assessment. 

b.  Some  terms  have  more  than  one  description;  when  this  is  the 
case,  the  descriptions  either; 

(1)  Are  significantly  different  between  sources  (though  the 

effective  meaning  may  be  not  much  different) 

(2)  Are  used  differently  (different  users  or  technical  lan¬ 
guage) 

(3)  May  be  found  wi'thin  the  context  of  a  different  source 

(4)  Have  real  differences  in  meaning. 

Both  DoD  and  non-DoQ  (e.g.,  FIPS  PUBs,  NBS  Special  Publ ications ) 
sources  are  used.  The  non-DoD  sources  and  terms  are  not  mandated  for 
our  use,  but  are  rather  included  for  breadth  of  understanding,  for 
those  relevant  terms  commonly  used  with  .the  non-DoD  governmental 

and/or  private  sectors. 

c.  The  source  of  each  description  is  indicated  by  a  symool  in 

parentheses  before  that  source's  term  description:  * 
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The  symbols 

(AF0TECP1) 

(AF0TECP3) 

(AFR800-14) 

(AFR300-L5) 

(AF0TECP5) 

(ROWE) 

(AFR205X) 

(CURRENT) 


( SYMBOL L  L) 
Description^  ^ 
(SYMB0L1  2) 
Desorption.  . 

x  •  C 


(SYMBOL^) 
Description,  ... 

i  •  n 


TERM 


N 


usjed  and  corresponding  sources  are: 


AFQTECP  8 00-2,  Volume  I,  10  Nov  92,  "Software  Test 
Manager-1  s  Guide.1' 

AFOTECP  800-2,  Volume  III,  1  Jan  84,  "Software  Main¬ 
tainability  Evaluator's  Guide." 

Air  Force  Regulation  300-14,  Volume  I,  'Management  of 
Computer  Resources  in  Systems,"  12  Sep  75. 

Air  Force  Regulation  300-15,  "Automated  nata  System 
Project  Management,"  Jan  78. 

AFOTEC  80-2,  Volume  5,  25  Jul  83,  "Software  Support 
Facility  Eva1uation--User 1  s  Guide." 

Rowe,  William,  An  Anatomy  of  Risk,  John  Wiley,  1977. 

Air  Force  Regu]ation  205-16,  "Automatic  Oata  Process¬ 
ing  (AOP)  Security  Policy,  Procedures  and  Responsi¬ 
bilities,"  1  Aug  84. 

Current  document  definition.  • 
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B.2  GLOSSARY  OF  TERMS  FOR  DEVELOPING  AND  IMPLEMENTING  A  RISK 
ASSESSMENT  METHODOLOGY  FOR  SOFTWARE  SUPPORTABILITY . 

Application  Software 

(AF0TECP5) 

The  software  written  by  software  support  personnel,  or  purchased 
from  a  contractor,  used  directlyin  supporting  ECSs.  It  is  nor¬ 
mally  used  for  simulation,  testing,  and  ECS  code  development. 

Application  Software  (functional) 

(AFR205X) 

Those  routines  and  programs  designed  by  or  for  automatic  data  pro¬ 
cessing  system  users  and  customers  to  complete  specific,  mission- 
oriented  tasks,  jobs,  or  functions,  using  available  automated  data 
processing  equipment  and  basic  software.  Application  Software  may 
be  either  general  purpose  packages,  such  as  demand  deposit 
accounting,  payroll,  machine  tool  control,  etc.,  or  specific 
application  programs  tailored  to  complete  a  single  or  limited 
number  of  user  functions  (for  example,  base  level  personnel,  depot 
maintenance,  aircraft,  missile  or  satellite  tracking,  command  and 
control ,  etc.).  Except  for  general  purpose  packages  acquired 
directly  from  software  vendors  or  from  the  original  equipment 
manufacturers,  this  type  of  software  is  generally  developed  by  the 
user,  either  with  in-house  resources  or  through  contract  services. 

Automated  Software  Development  Tool 

(AF0TECP5) 

A  component  of  System  Software  that  assists  in  the  design,  imple¬ 
mentation,  documentation,  and  verification  of  ECS  software. 

Availability 

(AFR800-14) 

A  measure  of  the  degree  to  which  an  item  is  in  the  operable  and 
commitable  state  at  the  start  of  the  mission,  when  the  mission  is 
called  for  at  an  unknown  (random)  point  in  time.  (MIL- STD -721) 

(AF0TECP5) 

The  probability  that  a  system  is  operating  satisfactorily  at  any 
point  in  time  when  used  under  stated  conditions. 

Available  Person  Time  (APT) 

(CURRENT) 

The  software  support  person-months  available  for  a  particular 
software  release  computed  as  the  product  of  the  release  duration 
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in  months,  the  number  of  support  personnel,  and  the  percentage  of 
the  time  those  personnel  are  dedicated  to  the  subject  software 
release  (versus  shared  across  other  releases  or  other  software 
systems).  This  time  includes  overhead  activity  directly  related 
to  the  subject  release.  The  release  duration  is  the  release  engi¬ 
neering  completion  date  minus  the  release  start  date. 

Basel i ne 

(AFR300-15) 

A  configuration  identification  document  or  set  of  such  documents 
formally  designated  and  fixed  at  a  specific  time  during  a  CPC  I ' s 
life  cycle.  Baselines,  plus  approved  changes  to  those  baselines 
constitute  the  current  conf iguration  identification. 

(ROWE) 

A  known  reference  used  as  a  guide  for  further  develooment  activ¬ 
ities. 

Baseline  Profile 
(CURRENT) 

See  Baseline  Software  Supportabi 1 ity  Profile. 

Baseline  Software  Supportabi 1 ity  Agreement 
(CURRENT) 

The  software  support  concept  and  baseline  software  supportabi 1 i ty 
profile  agreed  upon  by  tne  user  fusing  command)  and  the  supporter 
( support i ng  command ) . 


Baseline  Software  Supportao i ’ i ty  Profile 


8 


(CURRENT) 

The  set  of  27  pairs  of  numbers  (or  any  subset)  determined  by  spe¬ 
cifying  the  (time  to  complete  request,  number  of  requests  per  unit 
time)  pair  for  each  request  category.  A  request  category  is  the 
triple  (type,  priority,  complexity)  where  type  is  conversion, 
enhancement,  or  correction:  priority  is  emergency,  urgent,  or  nor¬ 
mal;  and  complexity  is  high,  medium,  low.  The  time  to  complete 
request  may  be  integrated  as  the  time  for  a  release,  and  specified 
requests  become  the  content  of  the  release. 

Block  Release 


(CURRENT) 

See  Release. 
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(AFR800-14) 

A  series  of  instructions  or  statements  in  a  form  acceptable  to  an 
electronic  computer,  designed  to  cause  the  computer  to  execute  an 
operation  or  operations. 

Computer  Resources 

(AFR800-14) 

The  totality  of  comDuter  equiDme.nt,  computer  programs,  associated 
documentation,  contractual  services,  personnel  and  supplies. 

Configuration  Control 

(AFR300-15) 

The  systematic  evaluation,  coordination,  approval  or  disapproval, 
and  implementation  of  approved  changes  in  the  configuration  of  a 
CPC  I  after  formal  establishment  of  its  conf iguration  identifica¬ 
tion. 

Configuration  [tern  (Cl) 

(AFR3CO-15) 

An  item  of  ADPE  that  is  designated  for  conf iguration  management. 

( 4FR800- 14) 

An  aggregation  of  equioment/software,  or  any  of  its  discrete  por¬ 
tions,  *n icn  satisfies  an  end  use  function  and  is  designated  oy 
tne  Government  for  conf iguration  manageme't.  CIs  may  vary  «’oe'y 
in  ccmciex’ty,  s  i  Z°  and  type,  from  an  a’r-,-aft  or  electronic  s,s- 
tem  to  a  test  meter  or  round  of  ammun:t;on.  Curing  deve'cpment 
and  init’al  production,  CIs  are  only  these  specification  ’tens 
tnat  are  referenced  d’rectly  in  a  contract  'or  an  ecu 1 va ' e^t 
in-nouse  agreement ) .  3u  r  •  ng  the  operation  and  maintenance  per-;:, 
any  repairable  item  designated  fo r  separate  procurement  -s  a  ten- 
figuration  item  (AFR  65-3). 

Configuration  Management  1C M) 

(AFR  300- 15) 

A  management  discipi ine  tnat  applies  tec"n’ca'  anq  aom- " ■ str it  -  .e 
direction  ana  surve ’ ’ : ance  to: 

(1)  Identify  and  document  the  funct;cna'  anq  prys'-ca'  :"ara:- 
teris;t;cs  pf  a  ;orf iguration  item 

(2)  .Control  changes  to  tnose  character ’ st ' cs 

(3)  Record  and  ro;.;rt  tonf  i  gurat  i  on  states. 
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Degree  of  Uncertainty 
(ROWE) 

That  proportion  of  information  about  a  total  system  that  is 
unknown  in  relation  to  the  total  information  about  the  system. 

Descriptiveness 

(CURRENT) 

A  measure  of  the  extent  that  software  products  contain  information 
regarding  its  objectives,  assumptions,  inputs,  processing,  out¬ 
puts,  components,  revision  status,  etc. 

Descriptive  Uncertainty 

(ROWE) 

The  absence  of  information  about  the  completeness  of  the  descrip¬ 
tion  of  the  degrees  of  freedom  of  a  system. 

Documentation 

(AF0TECP5) 

All  of  the  written  work  describing  operating  and  maintenance  pro¬ 
cedures  for  a  system. 

Documentation  Consistency 

(AF0TECP5) 

A  measure  of  the  consistency  in  the  information  provided  in  sup¬ 
port  system  documentation. 

Documentation  Oescriptiveness 

(AF0TECP5) 

A  measure  of  the  descriptiveness  of  the  information  provided  in 
support  system  documentation. 

Documentation  Modularity 

(AF0TECP5) 

A  measure  of  the  modular  organization  of  information  provided  in 
support  system  documentation. 

Documentation  Simplicity 

(AF0TECP5) 

A  measure  of  the  ease  of  use  and  lack  of  complexity  in  the  infor¬ 
mation  provided  in  computer  system  documentation. 
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Embedded  Computer  Resources 
(AF0TECP1) 

Computer  resources  incorporated  as  integral  parts  of,  dedicated 
to,  required  for  direct  support  of,  or  for  the  upgrading  or  modi¬ 
fication  of  major  or  less  than  major  system(s)  '(excludes  ADP 
resources  as  defined  and  administered  under  AFR  3C0  series) 
(USAF/RD/LE  Policy  letter,  13  October  1981). 

Embedded  Computer  System  (ECS) 

(AF0TECP1) 

a)  A  computer  that  is  integral  to  an  electromechanical  system  and 
that  has  the  following  key  attributes: 

(1)  Physically  incorporated  into  a  large  system  wnose  primary 
function  is  not  data  processing 

(2)  Integral  to,  or  supportive  of,  a  larger  system  from  a 
design,  procurement,  and  operations  viewpoint 

(3)  Inputs  include  target  data,  environmental  data,  command 
and  control,  etc. 

(4)  OutDuts  include  target  information,  flight  information, 
control  signals,  etc. 

b)  In  general,  an  embedded  computer  system  (ECS)  is  deve'oped, 
acquired,  and  operated  under  decentralized  management  (DoD 
Directives  5000.1,  5000.2). 

(AF0TECP5) 

A  computer  that  is  integral  to  an  electronic  or  electromechanical 
system  (e.g.,  aircraft,  missile,  spacecraft,  communications 
device)  from  a  design,  procurement,  and  operational  viewpoint. 

Emergency  MA 

(CURRENT) 

See  Maintenance  Priority. 

Enhancement  (Perfective)  MA 
(CURRENT) 

See  Maintenance  Type. 

Estimation 

(ROWE) 

The  assignment  of  probability  measures  to  a  postulated  future 

event. 
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Estimator  Uncertainty 
(ROWE) 

Uncertainty  in  measurement  resulting  from  deliberate  use  of  less 
complex  measures  such  as  central  value  estimates  of  dispersion  and 
smoothing  functions  for  time-dependent,  parameters. 

Evaluation 

(ROWE) 

Comparison  of  an  activity  performance  with  the  objectives  of  the 
activity  and  assignment  of  a  success  measure  to  that  performance. 

Evaluation  Criteria 

(AF0TECP1) 

Standards  by  which  achievement  of  required  operational  effective¬ 
ness/suitability  characteristics  or  resolution  of  technical  or 
operational  issues  may  be  judged.  For  full-scale  development  and 
beyond,  evaluation  criteria  must  include  quantitative  goals  (the 
desired  value)  and  thresholds  (the  value  beyond  which  the  charac¬ 
teristic  is  unsatisfactory)  whenever  possible  (OoD  Directive 
5000.3). 

Expandability 

(CURRENT) 

A  measure  of  the  extent  that  a  physical  change  to  information, 
computational  functions,  data  storage,  or  execution  time  can  be 
easily  accomplished  once  the  nature  of  what  is  to  be  changed  is 
understood . 

(AF0TECP5) 

A  measure  of  the  ease  with  which  the  functional  capability  of  com¬ 
puter  hardware  or  software  may  be  expanded. 

Faci 1  ity 

(AF0TECP5) 

The  physical  plant  and  the  services  it  provides;  specific  examples 
are  physical  space,  electrical  power,  physical  and  electromagnetic 
(TEMPEST)  security,  environmental  control,  fire  safety  provisions, 
and  communications  avai iaoi i i ty. 

Feedback 

(ROWE) 

The  return  of  performance  data  to  a  point  permitting  comparison 
with  objective  data,  normally  for  the  purpose  of  improving  per¬ 
formance  (goal-seeking  feedback),  but  occasionally  to  modify  the 
objective  (goal -changing  feedback). 
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F i rmware 
(AF0TECP1) 

a)  Computer  programs  and  data  loaded  in  a  class  of  memory  that 
cannot  be  dynamically  modified  by  the  computer  during  processing. 

b)  Hardware  that  contains  a  computer  program  and  data  that  cannot 
be  changed  in  its  application  environment. 

Note  1.  Computer  programs  and  data  contained  in  firmware  a^e 
classified  as  software;  the  circuitry  containing  the  computer  pro¬ 
gram  and  data  is  classified  as  hardware  (Data  and  Analysis  Center 
for  Software). 

High  Complexity  MA 

(CURRENT) 

See  Maintenance  Complexity. 

Historical  Maintenance  Profile 
(CURRENT) 

A  histogram  of  data  on  software  system  releases,  with  the  x-axis 
representing  discrete  ranges  of  (available)  person-months  per 
change  and  the  y-axis  representing  the  number  of  software  system 
releases  that  fall  into  each  x-axis  discrete  range.  For  purposes 
of  analysis  or  illustration,  the  axec  may  oe  re.e-'sec. 

Independent  Verification  and  Validation  (IV&V) 

(AF0TECP1) 

An  indeoendent  assessment  orocess  structured  to  ensure  that  com¬ 
puter  programs  fulfill  the  requirements  stated  in  system  and  sub¬ 
system  specifications  and  satisfactorily  perform  the  functions 
required  to  meet  the  user's  and  supporter's  requirements.  IV&V 
consists  of  three  essential  elements:  independent,  verif ication, 
and  validation: 

(1)  Independent.  An  organization/agency  which  is  separate 
from  the  software  development  activity  from  a  contractual 
and  organizational  standpoint. 

(2)  Verification.  The  evaluation  to  determine  whether  the 

products  of  each  step  of  the  computer  program  development 
process  fulfill  all  requirements  levied  by  the  previous 
step. 

(3)  Validation.  The  integration,  testing,  and/cr  evaluation 
activities  carried  cut  at  the  system/subsystem  level  to 
evaluate  the  developed  computer  program  against  the 
system  specifications  and  the  user's  and  supporter's 
requirements  (AFR  38-14). 
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Individual  Risk  Evaluation 


(ROWE) 

The  complex  process,  conscious  or  unconscious,  whereby  an  individ¬ 
ual  accepts  a  given  risk. 

Initial  Operational  Capability  (IOC) 

(CURRENT) 

That  point  in  a  system's  life  cycle  when  the  agreed  upon  number  of 
production  systems  has  been  delivered  to  the  user  (using  command) 
for  operational  use. 

Instrumentation 


(CURRENT) 

A  measure  of  the  extent  that  software  products  contain  aids  that 
enhance  testing. 

Interoperabi 1 ity 

(AF0TECP5) 

A  measure  of  the  degree  to  which  computer  hardware/soft^are  can 
interface  to  and  operate  with  other  similar  compute'- 
hardware/software. 


Low  Complexity  MA 
(CURRENT) 

See  Maintenance  Complexity. 

Maintainabil ity 
(AF0TECP5) 

The  probability  that  a  system  out  of  service  for  maintenance  can 

be  properly  repaired  and  returned  to  service  in  a  stated  elapsed 

time. 

Maintenance  Complexity 
(CURRENT) 

The  general  degree  of  difficulty  to  complete  a  maintenance 

request:  high,  medium,  low. 

High:  An  MA  where  cnanges  are  ;n  requirements,  design,  code,  and 
test;  or  greater  than  10%  of  CSCI  is  affected;  or  several  modules 
are  affected  by  the  change  (global  changes);  or  the  technical 

nature  of  the  change  requires  highly  specialized  personnel  skills; 
or  the  level  of  effort  by  personnel  is  large. 
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Medium:  An  MA  where  changes  are  in  design,  code  and  test;  or 
between  1%  and  10%  of  CSC1  is  affected;  or  at  least  two  modules 
are  affected  by  the  change  (semi-local);  or  the  level  of  effort  by 
personnel  is  average. 

Low:  An  MA  where  changes  are  isolated  to  only  one  unit  (e.g.,  one 
module/compilation  unit)  of  code:  or  no  more  than  1%  of  CSCI  is 
affected;  or  the  level  of  effort  by  personnel  is  minimal. 

Maintenance  Documentation 

(AF0TECP5) 

The  documentation  that  descr-'bes  the  maintenance  of  compute'-  sys¬ 
tem  hardware  and  software. 

Maintenance  Priority 

(CURRENT) 

The  criticality  of  the  maintenance  request  in  order  to  preserve 
mission  readiness:  emergency,  urgent,  normal. 

Emergency:  An  MA  requiring  all  available  personnel's  dedicated 
effort  to  correct  the  problem  as  soon  as  possible  (e.g., 
24  hours);  M I L-STD- 1679  severity  code  1  or  2:  mission  termination 
or  severe  degradation. 

Urgent:  An  MA  requiring  next  "block  release"  turnaround;  MIL-STD- 
1679  severity  code  3:  mission  impact. 

Normal:  An  MA  not  in  the  Emergency  or  Urgent  categories:  MIL-STD- 
1679  severity  code  4  or  5:  mission  inconvenience. 

Maintenance  Profile 

(CURRENT) 

See  Historical  Maintenance  Profile. 

Maintenance  Request  Category 
(CURRENT) 

The  identification  of  a  maintenance  request  by  specification  of 
the  maintenance  priority,  type,  and  complexity. 

Maintenance  Type 


(CURRENT) 

The  type  of  maintenance  actions  required  to  complete  a  maintenance 
request:  conversion,  enhancement,  correction. 
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Conversion  (Adaptive)  MA:  Any  change/effort  to  a  software  system 
which  is  initiated  as  a  result  of  changes  in  the  environment 
(e.g.,  hardware,  system  software)  in  which  the  software  system 

must  operate. 

Enhancement  (Perfective)  MA:  Any  change,  insertion,  deletion, 
modification,  or  extension  made  to  a  software  system  to  meet  the 
evolving  needs  of  the  user. 


Corrective  MA:  Any  change  which  is  necessitated  by  actual  faults 

(induced  or  residual)  in  a  software  system.  •’] 


Measured  Risk  Level 


(ROWE) 

The  historic,  measured,. or  modeled  risk  associated  with  a  given 
activity. 

Measured  Uncertainty 
(ROWE) 

The  absence  of  information  about  the  specific  value  of  a  measur¬ 
able  variable. 


Medium  Complexity  MA 
(CURRENT) 

See  Maintenance  Complexity. 

Modularity 

(CURRENT) 

A  measure  of  the  extent  that  a  logical  partitioning  of  software 
products  into  parts,  components,  and/or  modules  has  occurred. 


Module 


(AFR300-15) 

A  program  unit  that  is  discrete  and  identifiable  with  respect  to 
compiling  and  combining  with  other  units. 


» 
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Normal  MA 
(CURRENT) 

See  Maintenance  Priority. 
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Operational  Effectiveness 
(AF0TECP1) 

The  overall  degree  of  mission  accomplishment  of  a  system  used  by 
representative  personnel  in  the  context  of  the  organization,  doc¬ 
trine,  tactics,  threat  (including  countermeasures  and  nuclear 
threats),  and  environment  in  the  planned  operational  employment  of 
the  system  (OoO  Directive  5000.3). 

Operational  Suitability 

(AFOTECPI) 

The  degree  to  wnicn  a  system  can  be  satisfactory ly  placed  in  field 
use,  with  consideration  oeing  given  avail aD i 1 i ty ,  compatioi i i ty , 
transportabi 1 ity,  interoperabi 1 ity ,  reliability,  wartime  usage 
rates,  maintainability,  safety,  human  factors,  manpower  support- 
ability,  logistic  supportabi 1 ity,  and  training  requirements  (DoO 
Directive  5000.3). 

Opinion  Survey/Sampling 

(ROWE) 

Any  procedure  for  obtaining  by  oral  or  written  i nterrogation  or 
both  the  views  of  any  portion  of  the  affected  population  regarding 
benefit  levels  exoected,  their  utility,  and/or  relative  impor¬ 
tance.  Typically,  scientific  sampling  procedures  would  be  used  to 
maximize  (for  a  given  'evel  of  effort)  the  accuracy  ana  precision 
of  the  results  obtained. 

Parametric  Variation 

(ROWE) 

A  technique  for  sensitivity  analysis  of  any  given  model  in  whicn 
the  values  of  parameters  that  are  input  to  the  model's  calculation 

are  systematically  varied  to  permit  observation  of  how  sucn  vari¬ 
ation  affects  the  model's  output  (especially  ranking  of  alterna¬ 
tives)  . 

Person-Months  per  Change  (PMPC) 

(CURRENT) 

The  available  person-t'me  divided  by  the  total  numoer  of  cnange 
requests  for  a  software  system  release. 

Personnel 

(CURRENT) 

See  Support  Personnel. 
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Personnel  Skill  Level 
(CURRENT) 

A  subjective  integer  rating  from  1  (lowest)  to  5  (highest)  of 
software  support  personnel  experience,  education,  and  specific 
task  responsibility  capabilities. 

Probability 

(ROWE) 

A  numerical  property  attached  to  an  activity  or  event  whereby  the 
likelihood  of  its  future  occurrence  is  expressed  or  clarified. 

Probability  Distribution 

(ROWE) 

The  representation  of  a  repeatable  stochastic  process  by  a  func¬ 
tion  satisfying  the  axioms  of  probability  theory. 

Probability  of  Occurrence 

(ROWE) 

The  probability  that  a  particular  event  will  occur,  or  will  occur 
in  a  given  interval . 

Probability  Threshold 

(ROWE) 

A  probability  of  occurrence  level  for  a  risk  below  which  a  risk 
agent  is  not  long  concerned  with  the  risk  and  ignores  it  in  prac¬ 
tice  (threshold  of  concern). 

Product  Baseline 

(AFR300-15) 

The  initial  approved  product  configuration  identification. 

Program  Management  Responsibility  Transfer  (PMRT) 

(AFR800-14) 

That  point  in  time  when  the  designated  Supporting  Command  accepts 
program  management  responsibilities  from  the  Implementing  Command. 
This  includes  logistic  support  and  related  engineering  and  pro¬ 
curement  responsibilities  (AFR  800-4). 

Program  Support  Tools 

(AF0TECP3) 

General  debug  aids,  test/retest  software,  trace  software/hardware 
features,  use  of  compi ler/1 ink  editor,  library  management/configu¬ 
ration  management/text  editor/display  software  tools. 
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(AF0TECP3) 

Set  of  descriptions  and  procedures  for  how  the  program  is  to  be 
(or  can  be,  or  has  been)  tested. 

Propensity  for  Risk  Acceptance 

(ROWE) 

An  individual,  subjective  trait  designating  the  degree  of  risk  one 
is  willing  to  subject  oneself  to  for  a  particular  purpose. 

Qua! ity  Assurance  (QA) 

(AFR300-15 ) 

All  actions  that  are  taken  to  assure  that  a  development  organi¬ 
zation  delivers  products  that  meet  performance  requirements  and 
adhere  to  standards  and  procedures. 

Release 

(CURRENT) 

A  version  of  a  software  system  representing  either  the  initial 
baseline  configuration  or  an  update  to  a  previous  version  that 
incorporates  a  defined  set  of  software  change  requests.  Each 
release  becomes  a  new  baseline  configuration. 

Release  Engineering  Completion  Date 

(CURRENT) 

The  date  when  the  software  engineering  activity  for  a  release  is 
complete.  The  software  engineering  activity  includes  configura¬ 
tion  management,  quality  assurance,  and  software  maintenance  pro¬ 
ject  phases  of  requirements ,  design,  code,  unit  test,  integration 
test,  and  operational  test.  Activity  including  "kit  proofing," 
prom  burning,  and  in  general  technical  order  modifications  which 
typically  occur  between  engineering  completion  and  field  imple¬ 
mentation  (distribution)  is  not  included. 

Release  Field  Date 

(CURRENT) 

The  date  when  a  software  system  release  is  officially  distributed 
and  implemented  in  the  field  for  operational  use. 

Release  Id 

(CURRENT) 

A  unique  identifier  for  a  software  system  release. 
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Release  Start  Date 
(CURRENT) 

The  date  when  major  analysis  activity  related  to  a  specified 
release  begins  for  which  software  support  resources  are  required. 

Rel iabi 1 ity 

(ROWE) 

The  probability  that  the  system  will  perform  its  required  func¬ 
tions  under  given  conditions  for  a  specified  operating  time. 

Risk 

(ROWE) 

The  potential  for  realization  of  unwanted,  negative  consequences 
of  an  event. 

Risk  Acceptance 

(ROWE) 

Willingness  of  an  individual,  group,  or  society  to  accept  a  spe¬ 
cific  level  of  risk  to  obtain  some  gain  or  benefit. 

Risk  Acceptance  Function 

(ROWE) 

A  subjective  operator  relating  the  levels  of  probability  of  occur¬ 
rence  and  value  of  a  consequence  to  a  level  of  risk  acceptance. 

Risk  Acceptance  Level 

(ROWE) 

The  acceptable  probability  of  occurrence  of  a  specific  consequence 
value  to  a  given  risk  agent. 

Risk  Acceptance  Utility  Function 

(ROWE) 

The  profile  of  the  acceptability  of  the  probability  of  occurrence 
for  all  consequences  involved  in  a  risk  situation  for  a  specific 
risk  agent. 

Risk  Agent 


(ROWE) 

See  Valuing  Agent. 
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Risk  Assessment 


(AFR205X) 

A  detailed  study  of  the  vulnerabilities,  threats,  likelihood,  loss 
or  impact,  and  theoretical  effectiveness  of  security  measures. 
The  results  of  a  risk  assessment  may  be  used  to  develop  security 
requirements  and  specifications. 


;rowe) 


The  total  process  of  quantifying  a  risk  and  finding  an  acceptable 
level  of  that  risk  for  an  individual,  group,  or  society.  It 
involves  both  risk  determination  and  risk  evaluation. 


Risk  Assessment  Methodology  for  Software  Supportabi 1 i ty  (RAMSS) 


(CURRENT) 

A  method  of  determining  the  disparity  between  the  acceptable  risk 
(determined  from  the  support  concept,  baseline  software  support- 
ability  profile,  and  historical  maintenance  profile)  and  the  mea¬ 
sured  risk  (determined  from  a  conversion  of  the  software  supporta- 
bility  evaluation  metrics). 


Risk  Aversion 


(ROWE) 

The  act  of  reducing  risk, 


Risk  Baseline 


(CURRENT) 

The  risk  probability  density  function  and  the  associate  magnitude 
of  consequence  for  the  potential  negative  outcomes. 


Risk  Consequence 


(ROWE) 

The  impact  to  a  risk  agent  of  exposure  to  a  risky  event. 


Risk  Conversion  Factor 


(ROWE) 

A  numerical  weight  allowing  one  type  of  risk  to  be  compared  to 
another  type. 


Risk  Determination 


(ROWE) 

The  process  of  identifying  and  estimating  and  magnitude  of  risk. 
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I*  Risk  Estimation 

(ROWE) 

The  process  of  quantification  of  the  probabilities  and  consequence 
values  for  an  identified  risk. 

Risk  Evaluation 

(ROWE) 

The  complex  process  of  developing  acceptable  levels  of  risk  to 
individuals  or  society. 

Risk  Evaluator 

(ROWE) 

A  person,  group,  or  institution  tnat  seeks  to  interpret  a  valuing 
agent's  risk  for  a  particular  purpose. 

Risk  Identification 

(ROWE) 

The  observation  and  recognition  of  new  risk  parameters,  or  new 
relationships  among  existing  risk  parameters,  or  perception  of  a 
change  in  the  magnitude  of  existing  risk  parameters. 

!  Risk  Profile  3aseline 

(CURRENT) 

The  measure  of  information  and/or  requirements  which  serve  as  the 
zero  reference  against  which  negative  (and  positive)  outcomes  can 
be  determined. 

Risk  Reduction 

(ROWE) 

The  action  of  lowering  the  probability  of  occurrence  and/or  tne 
value  of  a  risk  consequence,  thereby  reducing  the  magnitude  of  the 
risk. 

Risk  Reference 
(ROWE) 

Some  reference,  absolute  or  relative,  against  which  the  accept¬ 
ability  of  a  similar  risk  may  be  measured  or  elated;  implies  some 
overall  value  of  risk  to  society. 

Sensitivity  Analysis 

(ROWE) 

A  method  used  to  examine  the  operation  of  a  system  by  measuring 
J<>  the  deviation  of  its  nominal  behavior  due  to  perturbations  in  the 

performance  of  its  components  from  their  nominal  values. 
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(CURRENT) 

A  measure  of  the  extent  that  software  products  reflect  the  use  of 
singularity  concepts  and  fundamental  structures  in  organization, 
language,  and  implementation  techniques. 

Simulation 

(AFR800-14) 

The  representation  of  physical  systems  or  phenomena  by  computers, 
models  or  other  equipment. 


(CURRENT) 

A  software  support  site,  or  particular  location,  where  software 
support  activity  is  being  accomplished.  Includes  sites  such  as 
the  Air  Logistics  Centers  (ALCs). 

Site  Survey  Form 

(CURRENT) 

The  data  collection  form  used  during  the  software  support  site 
visits  to  collect  background,  evaluation,  and  maintenance  release 
data.  See  appendix  C. 


Software 

(AF0TECP1) 

A  set  of  computer  programs,  procedures,  and  associated  documenta¬ 
tion  concerned  with  the  operation  of  a  data  processing  system. 

(CURRENT) 

The  programs  which  execute  in  a  computer.  The  data  input,  output, 
and  controls  upon  which  program  execution  depends  and  the  documen¬ 
tation  which  describes,  in  a  textual  medium,  development  and  main¬ 
tenance  of  the  program. 

Software  Change  Request 

(CURRENT) 

An  official  request  that  could  involve  a  change  to  a  software  sys¬ 
tem.  Such  requests  include  problem  report,  enhancement  require¬ 
ment,  modification  request,  or  any  other  form  that  is  officially 
tracked  by  a  configuration  management  function. 
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Software  Configuration  Management 
(CURRENT) 

A  discipline  applying  technical  and  administrative  direction  and 
surveillance  to  1)  identify  and  document  the  functional  and  phys¬ 
ical  characteristics  of  a  configuration  item,  2)  control  changes 
to  those  characteristics  and  3)  record  and  report  change  process¬ 
ing  and  implementation  status. 

Software  Delivery 

(CURRENT) 

That  point  in  the  software  life  cycle  when  the  software  support 
function  assumes  responsibi 1 ity  for  the  "next"  set  of  configura¬ 
tion  changes  to  the  software  (e.g.,  next  block  release).  This 
point  is  logically  no  later  than  PMRT,  but  could  be  as  early  as 
IOC.  This  applies  when  a  contractor  or  government  agency  assumes 
the  software  support  function. 

Software  Error 

(CURRENT) 

The  human  decision  (inadvertent  or  by  design)  which  results  in  the 
inclusion  of  a  fault  in  a  software  product. 

Software  Fault 

(CURRENT) 

The  presence  or  absence  of  that  part  of  a  software  product  which 
can  result  in  software  failure. 

Software  Life  Cycle  Process  Management 

(CURRENT) 

The  policy,  methodology,  procedures,  and  guidelines  applied  in  a 
software  environment  to  the  software  development  and  support  life 
cycle  activities. 

Software  Maintainability 

(AF0TECP1) 

The  ease  with  which  software  can  be  changed  in  order  to: 

(1)  Correct  errors 

(2)  Add/modify  system  capabilities  through  software  changes 

(3)  Delete  features  from  programs 

(4)  Modify  software  to  be  compatible  with  hardware  changes. 
(CURRENT) 

A  quality  of  software  which  reflects  the  effort  required  to  per¬ 
form  software  maintenance  actions. 
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Software  Maintenance 
(CURRENT) 

Those  actions  required  for: 

(1)  Correction  -  Removal,  correction  of  software  faults 

(2)  Enhancement  -  Addition/deletion  of  features  from  the 
software 

(3)  Conversion  -  Modification  of  the  software  because  of 
environment  (data  hardware)  changes. 

Software  Maintenance  Environment 

(CURRENT) 

An  integration  of  personnel  support  systems  and  physical  facili¬ 
ties  for  the  purpose  of  maintaining  software  products. 

Software  Maintenance  Measures 

(CURRENT) 

Measures  of  software  maintainability  and  environment  capabilities 
to  support  software  maintenance  activity. 

Software  Maintenance  Project  Management 

(CURRENT) 

The  software  life  cycle  process  management  applied  during  the  sup¬ 
port  phase  for  the  software  to  accomplish  specific  software  main¬ 
tenance  tasks  which  derive  from  software  problem  reports  or  change 
requests. 

Software  Management 
(CURRENT) 

The  policy,  methodology,  procedures,  and  guidelines  applied  in  a 
software  environment  to  the  software  development/maintenance 
activities.  Also,  those  personnel  with  software  management  res¬ 
ponsibilities. 

Software  Reliability 

(CURRENT) 

A  quality  of  software  which  reflects  the  probability  of  failure 
free  operation  of  a  software  component  or  system  in  a  specified 
environment  for  a  specified  item. 

Software  Portability 

(CURRENT) 

A  quality  of  software  which  reflects  the  effort  required  to  trans¬ 
fer  the  software  from  one  environment  (hardware  and  system  soft¬ 
ware)  to  another. 
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Software  Support  Facility  (SSF) 

(AF0TECP5) 

The  facility  which  houses  and  provides  services  for  the  support 
systems  and  personnel  required  to  maintain  the  software  for  a  spe¬ 
cific  ECS. 

Software  Support  Personnel 
(CURRENT) 

See  Support  Personnel. 

Software  Support  Resources 
(CURRENT) 

The  totality  of  personnel,  systems,  physical  facilities,  and  cal¬ 
endar  time  that  are  used/consumed  during  a  software  support 
release  effort. 

Software  Supportabi 1 ity 

(CURRENT) 

A  measure  of  the  adequacy  of  personnel,  resources,  and  procedures 
to  facilitate: 

(1)  Modifying  and  installing  software 

(2)  Establishing  an  operational  software  baseline 

(3)  Meeting  user  requirements. 

Software  Supportabi 1 ity  Evaluation 
(CURRENT) 

An  evaluation  to  derive  a  measure  of  how  well  a  software  system 
can  be  supported.  (See  Software  Supportabi 1 ity. ) 

Software  Supportabi 1 ity  Evaluation  Metrics 

(CURRENT) 

The  closed-form  questionnaire  scores  for  each  software  support- 
ability  characteristic  in  a  software  supportabi 1 ity  evaluation  as 
well  as  the  values  computed  by  cumulating  lower  level  scores. 

Software  Supportabi 1 ity  Magnitude  Of  Risk  Consequence 

(CURRENT) 

The  level  of  impact  to  a  software  user  or  supporter  as  a  result  of 
the  risk  level  of  a  software  supportabi 1 ity  negative  outcome. 

Software  Supportabi 1 ity  Negative  Outcome 

(CURRENT) 

Any  outcome  for  which  the  software  support  resources  are  not 
adequate  to  accomplish  required  software  support. 
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Software  Supportabi 1 ity  Risk 
(CURRENT) 

The  probability  at  a  given  point  during  the  software  support  phase 
that  the  software  maintenance  activity  specified  by  a  baseline 
software  supportabi 1 ity  profile  can  not  be  accomplished  with  the 
available  software  support  resources. 

Estimated  Software  Supportabi lity  Risk:  An  estimate  of  the 

software  supportabi 1 ity  risk  determined  by  the  area  under  an 
historical  maintenance  profile  curve.  The  area  is  the  part  under 
the  curve  to  the  "rignt"  of  the  suoject  software's  available 
person-months  per  change  value  as  computed  from  the  estimated 
software  support  concept  and  baseline  software  supportaDi 1 i ty 
profile.  Numerically,  this  area  is  the  numDer  of  software  system 
releases  in  the  historical  maintenance  profile  with  an  available 
person-months  per  change  greater  than  the  subject  software's 
computed  person-months  per  change  divided  by  the  total  number  of 
software  system  releases  in  the  historical  maintenance  profile. 

Acceptable  Software  Supportabi 1 ity  Risk:  The  estimated  software 

supportabi 1 ity  risk  which  is  agreed  upon  by  the  user  (using 
command)  and  supporter  (supporting  command)  as  a  result  of  the 
baseline  software  supportabi 1 ity  agreement. 

Evaluated  Software  Supportabi 1 ity  Risk:  An  approximation  to  the 

software  supportabi 1 ity  risk  computed  from  the  software  support- 
ability  evaluation  metrics.  The  computation  is  derived  from  one 
or  more  of  the  three  models  described  in  section  4:  simple  linear 
conversion,  linear  regression,  and  factor  analysis  regression. 

Measured  Software  Supportabi 1 ity  Risk:  See  Evaluated  Software 

Supportabi 1 ity  Risk. 

Software  Supportabi 1 ity  Risk  Agent  Acceptance  Level 
(CURRENT) 

The  software  supportabi 1 i ty  risk  level  which  is  acceptable  to  a 
risk  agent. 

Software  Supportabi 1 ity  Risk  Level. 

(CURRENT) 

The  potential  for  realization  of  a  software  supportabi 1 ity  nega¬ 
tive  outcome. 

Software  System 


(CURRENT) 

A  set  of  software  (specifications,  programs,  and  data)  which  con¬ 
stitutes  a  well-defined  major  function  or  group  of  functions. 
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Typical  systems  include  avionics  OFP,  ground  based  communications, 
missile  guidance,  simulation,  threat  generator,  ATE,  and  electro¬ 
nic  warfare. 

Software  System  Type 

(CURRENT) 

One  of  seven  classifications  of  a  software  system's  primary  func¬ 
tional  mission:  ATD,  ATE,  C-E,  EW,  OFP,  SIM,  SUP. 

ATD:  Aircrew  Training  Device  or  Operational  Flight  Trainer  for 
training  and  support  of  an  operational  system,  usually  in  the  form 
of  a  mockup  simulator. 

ATE:  Automatic  Test  Equipment  software  to  support  the  testing  of 
hardware  units  under  test  (UUT),  create  and  maintain  the  environ¬ 
ment  where  the  test  software  may  be  used,  or  prepare/analyze/main¬ 
tain  test  software. 


C-E:  Communications-Electronics  software  for  command  and  control, 
communications,  surveillance  and  warning,  air  traffic  control, 
intelligence,  and  other  related  functions. 

EW:  Electronic  Warfare  software  that  involves  the  use  of  electro- 
„  magnetic  energy  and  performs  functions  either  separate  or  integral 
#  to  a  larger  airborne  or  ground  system. 


OFP:  Operational  Flight  Program  software/f irmware  that  is  inte¬ 
gral  to  an  onboard  aircraft  computer  system  including  navigation, 
flight  control,  fire  control,  weapon  delivery,  electronic  engine 
control,  and  heads-up  display. 

SIM:  Simulation  Software  not  included  as  part  of  the  ATD,  includ¬ 
ing  simulation  models. 

SUP:  Support  Software  including  application  support  software  and 
system  support  software  not  included  in  any  other  category. 

Specification 

(AFR300-15) 

A  document  that  describes  the  requirements  for  the  development  or 
acquisition  of  ADPE  and/or  software. 

Source  Code 

(CURRENT) 

The  form  of  the  program  code  in  its  source  language. 
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Standards 

(AF0TECP3) 

Procedures,  rules,  and  conventions  used  for  prescribing  disci¬ 
plined  program  design  and  implementation. 

Subjective  Probabilities 

(ROWE) 

The  assignment  of  subjective  weights  to  possible  outcomes  of  an 
uncertain  event  where  weights  assigned  satisfy  axioms  of  probabil¬ 
ity  theory. 

Support  Concept 

(CURRENT) 

The  software  support  concept  usually  specified  as  part  of  the 
CRISP  and  OS/CMP.  Also  includes  that  part  of  the  support  concept 
necessary  to  establish  the  acceptable  risk  from  a  baseline 
software  supportabi 1 i ty  profile:  standard  release  duration, 

number  of  support  personnel,  average  skill  level,  percentage  of 
personnel  dedicated  to  releases,  support  facility,  etc. 

Support  Facility 

(CURRENT) 

The  physical  facility  resources  that  must  be  available  for  the 
software  support  resources  to  accomplish  a  specific  task(s). 

Support  Personnel 

(CURRENT) 

A  general  term  for  personnel  (military,  DoD  civilian,  or  DoD  con¬ 
tractor)  whose  skills  are  necessary  to  directly  support  mission 
critical  system  software  maintenance.  Includes  but  is  not  limited 
to  management,  technical,  non-technical  support,  and  contractor 
personnel . 

Support  System 

(AF0TECP5) 

Any  automated  system  used  to  change,  test,  or  manage  the  configu¬ 
ration  of  ECS  software  and  associated  documentation.  Includes  but 
is  not  limited  to  Host  Processor,  Software  Bench,  Laboratory- Inte¬ 
grated  Test  Facility,  Operational-Integrated  Test  Facility,  and 
Configuration  Management  System. 

Support  System  Facility 


(AF0TECP5) 

The  facility  resources  that  must  be  available  for  the  software 
support  resources  to  accomplish  a  specific  task(s). 


THE  BOM  CORPORATION 


8DM/A-35-0510-TR 


System  Software 
(AF0TECP5) 

All  of  the  software  that  is  part  of  the  software  support  facility 
computer  system.  It  is  never  or  seldom  accessed  directly  by  soft¬ 
ware  support  facility  personnel;  it  controls  the  processing  of 
aoplication  software.  It  includes  the  Operating  System,  Source 
Code  Editor,  Language  Translator,  Link  Editor/Loader , 
Librarian/Fi le  Manager,  Data  3ase  Manager,  and  Automated  Software 
Development  Tool. 

Threshold 

(ROWE) 

A  discontinuous  change  of  state  of  a  parameter  as  its  measure 
increases.  One  condition  exists  below  the  discontinuity,  and  a 
different  one  above  it. 

Time  to  Complete  Maintenance  Request  (TC) 

(CURRENT) 

The  calendar  time  from  receipt  of  the  maintenance  request  by  the 
support  control  group  until  the  request  has  been  accepted  as  part 
of  an  operational  system  software  configured  release.  (This  does 
not  mean  the  configuration  is  released  or  distributed,  and  this 
time  does  not  include  this  additional  delay,  if  any.) 

Uncertainty 

(ROWE) 

The  absence  of  information;  that  which  is  unknown. 

Urgent  MA 
(CURRENT) 

See  Maintenance  Priority. 

User 

(AFR205X ) 

Any  persons  (organizations)  having  access  to  an  automatic  data 
processing  system  via  communication  through  a  remote  device  or 
allowed  to  submit  input  to  the  system  through  other  media  (for 
example,  tape  or  card  decks).  (Does  not  include  those  persons  or 
organizations  defined  as  customers.) 

Valuing  Agent 

(ROWE) 

A  person  or  group  of  persons  who  evaluates  directly  the  conse¬ 
quence  of  a  risk  to  which  he  is  subjected.  A  risk  agent. 
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Verification/Validation  (of  computer  programs) 

(AFR800-14) 

The  process  of  determining  that  the  computer  program  was  developed 
in  accordance  with  the  stated  specification  and  satisfactorily 
performs,  in  the  mission  environment,  the  function(s)  for  which  it 
was  designed. 

Weight  (structured  value  analysis) 

(ROWE) 

The  relative  importance  of  terms  in  a  model  expressed  as  a  decimal 
fraction;  weights  for  a  set  of  terms  add  to  unity. 

Weighting  Factor 

(ROWE) 

A  coefficient  used  to  adjust  variable  accuracy  to  a  subjective 
evaluation;  these  factors  are  usually  determined  through  surveys, 
Delphi  sessions,  or  other  formats  expressing  social  priorities. 
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